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ABSTRACT 



Assessing the Impact of office automation (OA) requires expertise 
in the generic aspects of evaluation and innovation adoption, combined 
with specialized knowledge of OA, There is an extensive literature on 
the two generic subjects, but no companion literature concerning the 
application of that knowledge to the unique case of OA, By providing 
that specialized information, this report assists the implementors of OA 
in two ways: it shows them how to monitor implementation efforts, thus 
providing feedback to facilitate adoption of OA technology; and it pro- 
vides guidance for measuring OA's impact on people and organizations. 

The report assumes an immediate impact of OA on the work groups 
where the technology is implemented, and a continually spreading effect 
from that locus of immediate use. Included in the report are discussions 
of: sources of data, methods of data collection, factors which affect 
implementation, and measures of impact. Special attention is given to 
measuring productivity changes that may result from the use of OA, 

A detailed appendix supplies a variety of examples which show how 
the variables discussed in the report were actually measured in applied 
settings . 
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EXECUTIVE SUMMARY 



This document provides a framework for assessing the implementation 
and impact of office automation technology within shore-based facilities 
of the United States Navy. It is intended to help the Navy make effec- 
tive use of any office automation (OA) it implements. The plan's basic 
goals are to help personnel to: 

• monitor the process of implementation; 

• assess the impact of OA use; 

• clarify relationships between implementation strategies, user 
acceptance, and impact; and 

• identify organizational changes that would facilitate the 
greatest possible exploitation of OA technology. 

The two levels of information are conveyed. On the first level, the 
concepts required to evaluate the implementation of OA technology are 
discussed. On the second level, specific examples are given of how those 
concepts were measured in actual studies. In order to avoid excessive, 
complicating detail in the presentation of concepts, the examples all 
appear in an appendix. 

Together, the body of the report and the appendix represent a de- 
tailed and comprehensive list of issues related to assessing the imple- 
mentation and impact of OA. However, the intention is not to advocate an 
all-or-nothing approach to these recommendations. Rather, the goal is 
to provide OA implementors with an overview of what can be done, so they 
can make informed choices about the areas they need to pursue for their 
specific situation. 

This report is not a complete treatise on evaluation or innovation 
adoption, but deals with special issues which must be considered when OA 
is the subject of an impact assessment. 

Four assumptions guide this report: 

• The process of implementing OA can be planned and managed; 

• The quality of that management will affect the value of OA's 
use; 

• Understanding the course of implementation entails an 
appreciation of interaction of three factors - the 
technology itself, the people who use the technology, and the 
setting where the technology is used; and 

• OA can affect organizations in different ways, and each way 
must be assessed if the value* of OA's contribution to organiza- 
tional functioning is to be maximized. 
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This report employs a model which views OA use as an ongoing process 
ti'at begins vith implementation, has an immediate impact on end-users 
and their work groups, and spreads its impact from the immediate locus 
of use. Throughout the process, feedback loops are operating; that is, 
the users of OA are influenced by the way their work affects others. 



MEASUREMENT STRATEGY 

To provide a context for the work, it would be useful to begin with 
an understanding of other efforts to assess the impact of OA. These ef- 
forts fall into two main categories - assessment of the nature of OA 
technology, and of its impact. In each category, a crucial distinction 
must be made between studies that deal with variables in a more general 
way, and those concerned with specifics. For example, in some cases OA 
is referred to as a global concept, without specification of particular 
applications or hardware; other studies do make these specifications. 
The same is true for impact studies. Respondents can be asked general 
questions about impact, or questions which specify particular aspects of 
work - communications, report quality, decision making, and so on. 

For economy and efficiency, special attention must be paid to 
archival data - that information which already exists as a result of the 
normal course of events, and does not have to be collected as part of a 
special effort. In addition, much useful data can be gathered with a 
small amount of extra effort during routine interaction with end-users. 
For example, useful data on training needs can be collected with a few 
simple questions each time a help center staff member gives advice to an 
end-user. 

It is useful to employ multiple sources of data to measure each 
concept, for two reasons: first, the multiple sources serve as validity 
checks on each other; and second, different measures cf impact may cast 
light on the size of an observed change. For example, archival data may 
show that reports are being written more clearly and contain more infor- 
mation, while another indicator of report quality - reader perception - 
may show no change. Such a finding would indicate that although OA has 
affected report quality, more effort is needed if the change is to have 
any practical significance. 

The ideal strategy for wording questions is to use the responses to 
open ended questions as a basis for constructing more probing short- 
answer questions, which capture all relevant information. "Ease of res- 
ponse" should be the primary consideration when placing a series of ques- 
tions in order. Questions which prompt the respondent to recall relevant 
information should serve as a prelude to questions which require value 
judgments on specific events in answer. 

Because value judgments on the importance of OA- induced changes are 
inescapable, data on these judgments should be formally incorporated 
into an assessment strategy. For example, consider the use of OA in the 
assignment of personnel. If the technology is used to help improve the 
assignment of scarce personnel to critical specialties, even a small 



y 10 



increase in efficiency could have very important benefits. However, the 
£AIB& flBfiimt of improvement in efficiency would be relatively unimportant 
when dealing with large numbers of qualified people being assigned to 
non-critical specialties* 

Special consideration should be given to the issue of productivity 
improvement, which is particularly important when studying OA impact. 
First, all accepted measures of productivity * whether affected by 0*. or 
not - must be catalogued, to provide a sense of the rang* of OA's effect 
on productivity. Next, the appropriateness of those measures must be 
considered. For example, standard measures may rate a group equally on 
its accomplishment of both high and low priority work, but OA implemen- 
tors may wish to differentiate between the two when measuring impact. 
Finally, data on any "value added 1 * component that OA may contribute to 
the quality of the work should ue collected. One aspect of "value add- 
ing" is the ability to carry out new types of work that could not be 
accomplished without OA. The creation of new, previously-unavailable 
information is another aspect of OA "value adding." 



DATA TO BE COLLECTED 



Several variables can be useful in explaining why an OA implementa- 
tion process seems to be succeeding or faili.ng. These include: the rea- 
sons for OA acquisition (need driven vs. technology driven); the actions 
of key players in the process; the number of key players involved; the 
structure and amount of support for end-use of OA; the amount of exper- 
tise available to users, the relative ability of implementation planning 
process to change in light of new circumstances, and incentives for peo- 
ple to use the technology. 

Successful implementation does not guarantee continued use, so it is 
important to collect information on the routinization of OA use. 
Routinization-related factors include such things a* organizing wor!: to 
require OA use, formal training procedures, and user friendliness. 

In addition to the above factors, understanding implementation re- 
quires an understanding of end-users and their work groups, because they 
represent the immediate context where the hard choices on OA use will be 
made. Information should be collected on how OA fits into work routines, 
and on users' satisfaction with the OA that's available to them; a des- 
cription of how the work groups ave constituted, and their function, 
should also be made. 
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Beyond understanding the implementation process, it is important to 
measure its consequences. Those fall into five categories: 

• amount of products or services produced by user groups, 

• quality of those services, 

• groups' ability to adapt to new circumstances, 

• possible changes in the group's role in the organization, 
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• communication patterns, and 



• consequences outside the work group. 

Each of these represent a unique type of change that OA can bring about 
within an organization. 

A final important consideration to keep in mind is that any given 
change nay be for Vrter or worse. It is, therefore, very important to 
construct questions * and to maintain a perspective * that keeps the 
assessment process open to potentially negative consequences as well as 
to positive ones. 



THE DATA COLLECTION PROCESS 

Once the format and the substance of the questions have been de- 
cided upon, a schedule for data collection must be established. The 
first step is determining when information will be collected. Natural- 
ly, the ideal situation is one where data can be collected both before 
and after OA implementation, thus allowing precise estimates of the 
change wrought by OA. But even in the absence of before -and- after com- 
parisons, the timing of data collection can be critical. While some 
data can be gathered at any convenient time (archival data, for example), 
others must be obtained within a narrow window of opportunity. A delay 
in collection is one major problem, as accurate recollections of specific 
circumstances are lost. But collecting data too early is also a problem; 
users may not have had time to adapt and reflect on changes. Too-early 
collection may show detrimental effects of OA when in fact, the long-run 
impact was positive. 

In the early stages of implementation it is important to collect 
data from as many people as possible, to insure a good information base 
from which more structured questions can be developed. As OA usage 
spreads, however, a sampling plan will be required. Six principles must 
underlie any such plan: 

• Any groups of particular interest to OA implemsntors must be 
included. 

• The remainder of the sample must be representative of the pop- 
ulation of interest. 

• Because some data will have to be collected early in the im- 
plementation process, respondents should be identified as soon 
as possible. 

* special effort should be made to include groups who have spe- 
* relationships with end-users. 

ecipients of OA use products must be included in impact 
aies. 



ERIC 



xii 

12 



• The amount of information required from any given population 
may increase to the point of overburdening any single 
representative of that group. In such cases, it may be advis- 
able to ask each respondent for only a portion of the required 
information, and rely on multiple respondents to supply all the 
necessary data. 



NETWORKING AS AN INNOVATION 

Networking and stand-alone workstations must be considered as a sep- 
arate innovation. Although the basic approach to their evaluation is 
the same, there are important differences in emphasis. For example, 
understanding computer- computer interaction may require analysis of a 
setting's mainframe environment, or of group processes related to agree- 
ments on file structures and data elements among grouped workstations. 
These issues are not critical for understanding the use of stand-alone 
workstations. These differences of emphasis apply to studies of both 
the implementation and impact of OA technology. 

In terms of implementation, networking will require much greater 
attention to the role of key players, and to the use of an adaptive plan- 
ning strategy. The reason for this shift is that networking brings about 
tighter linkages among parts of an organization, thus increasing the 
mutual influence of innovation on organizational process, and vice versa. 

As examples of the difficulties that may result from tightening 
organizational linkages, consider the following questions: (1) As a re- 
sult of networking, what might happen to relationships among groups who 
previously had unique functions, but which relied on each other for in- 
formation? (2) Once a group has gone to the trouble of setting up its 
own "private" databases, will it readily agree to participate in a new 
shared- file system that requires changes in their already-proven file 
structure? 

The shift from stand-alone to networked systems also has implica- 
tions for assessing outcome. Although the variables of interest remain 
the same (output, quality, adaptability, evolving roles, communication 
and organization-wide consequences) , there are shifts in the likelihood 
that changes will occur, and in the direction, and size of those changes. 

Likelihood of Change 

There is an increased risk that networked systems will have no im- 
pact because changes can only occur if a system is implemented, and or- 
ganizational difficulties increase the chance that effective implementa- 
tion won't happen. 

Direction of Change 

Linking workstations increases the dependence of different parts of 
the organization on each other, which thus 4 — reases the probability of 
undesirable effects for three reasons: Fi , technical malfunctions 
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will affect all grcUps linked to the system. Second, the more sharing 
of information, th'i greater the negative consequences of passing incor- 
rect information, or of disseminating error-prone programs. Third, since 
sharing increases the dependence of work- groups on each other, there are 
increased difficulties if one group does not do its job well. 

Size of Change 

Networked systems may result in greater beneficial change than would 
standing workstations. One reason is that networking may free many work 
groups from the tedious job of data entry, thus providing opportunities 
to accomplish other work. More important, networking provides a vast in- 
crease in available information, and much faster communication. 
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1. PURPOSE 



This document provides a framework for assessing the implementation 
and impact of office automation technology within shore-based facilities 
of the United States Navy. It is intended to help the Navy make effec- 
tive use of any office automation (OA) it implements. 

Several factors underscore the importance of making strenuous ef- 
forts to increase the efficiency of shore-based operations. 

• The 600 -ship Navy will place severe limitations on the number 
of people available for the complex shore duty tasks required 
to maintain the military readiness of a 500,000 person force. 

• Because of the frequent changes in jobs by almost all Navy 
personnel, efficiency suffers from a lack of continuity and 
institutional memory. 

• Effective organizations require personnel who have good working 
relationships with each other, both on formal and informal 
levels. Frequent job changes and the press of business threaten 
the development of such relationships. 

If appropriately implemented and supported, information technology 

can assist in achieving a hign level of organizational functioning in 

light: of the above-stated threats to efficient operations. Properly 

used, office automation will contribute to better decision making, 

greater coordination among functions, and more efficient operation of 

routine office activities. 



2. SCOPE OF THE EVALUATION PLAN 

This evaluation plan is constructed sc ^hat its overall goals will 
remain constant, while its range and depth of analysis parallel the ex- 
tent to which information technology is actually implemented. The basic 
goals of the project are to: 

• monitor the process of implementation; 

• assess the impact of OA use; 

• clarify relationships between implementation strategies, user 
acceptance, and impact; and 

• identify organizational changes that would facilitate the 
greatest possible exploitation of OA technology. 

Throughout, the plan calls for a continual flow of information to 
those charged with implementing OA. They will need to know how implemen- 
tation can be improved, what impact the technology is having, and how 
the effort can be justified in terms of its contribution to Navy readi- 
ness. 

The bulk of this report will emphasize the analysis of stand-alone 
workstations, because they represent the most likely situation for OA 
implementation. End-users will begin using their equipment immediately 
upon delivery,. without waiting for networking to become operational. 
Because networking is likely to arise at some point, special considera- 
tions related to networking are presented in a special section. 

The intention in this report is to convey information on two levels. 
On the first level, the concepts required to evaluate the implementation 
of OA technology, and the relationships among those concepts are dis- 
cussed. On the second level, specific ways to measure those concepts 
are detailed. An example of a statement on the general level would be: 
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"User support Is required if OA is to improve the quality of a work 
group's reporting." The second level would explain specifically what to 
look for in measuring these concepts. Briefly, we might define "quality 
of reporting" as plans which are based on more information, faster res- 
ponse time to requests, or better use of graphs and tables. (Much more 
on definitions of quality will come later.) 

Presenting information on these levels will give the reader a sense 
of how evaluation efforts should first be conceptualized, then put into 
operation. In order to avoid excessive complicating detail in the pre- 
sentation of concepts, examples of specific measurement are presented in 
Appendix A. That appendix is drawn from actual studies of the impact of 
information technology. If a real example of measuring a concept men- 
tioned in this report could be found, that example was included in the 
appendix. 

It is beyond the scope of this report to present complete details 
on the measurement of each concept. There will, however, be at least 
one example of measurement for each concept. These examples can be used 
as they appear, or they can be viewed as a lesson in how to move from 
abstract concepts about the evaluating information technology, to the 
specifics of assessment. 

It is not our intention to argue that all recommendations in this 
report be implemented in an all-or-nothing fashion. Elements can be 
chosen as needed, and choices can be made about the value of information 
from more or less complex evaluations. For example, one could monitor 
implementation without assessing impact, or vice versa. Leaving out one 
or the other may result in less insight into the contribution of 
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information technology, but each type of information still has value in 
its own right. 

It is also not our intention to present a complete treatise on eval- 
uation, including topics of research design, instrument construction and 
statistical analysis. Such a presentation is far beyond the scope of 
this report, and would merely repeat information contained in many other 
sources. Rather, the intent is to remain close to the special factors 
which must be considered when assessing the implementation and impact of 
information technology. For basic information on generic issues in eval- 
uation, the following are recommended: introductions and overviews of 
evaluation methodology - Rossi and Freeman (1985) ; and qualitative evalu- 
ation methods - Guba and Lincoln (1981), or Patton (1978); survey and 
questionnaire methods - Dillman (1978). 

Throughout, we assume the reader does not have a deep background in 
evaluation; consequently, technical matters are discussed in lay terms. 
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3. BASIC ASSUMPTIONS 

Several basic assumptions guided the evaluation plan outlined in 
this report. 

1. The process of implementing information technology can be 
planned and managed. It need not be a haphazard process. 

2. The nature of that management will make a difference in terms 
of whether the technology is used, what advantages come from its use, 
what problems will arise, and how well those problems can be overcome. 
For example, many different strategies for training can accompany the 
arrival of OA. Users can be left to their own devices, electronic tutor- 
ials can be provided, help centers can be established, or formal training 
can be made available. Further, the timing of these types of training 
can be varied relative to the implementation of the technology. Explicit 
decisions about these issues can be made, and those decisions will affect 
the course of OA use. 

3. Understanding the course of implementation involves an appreci- 
ation of interactions among three factors - the nature of the technology, 
the people using the technology, and the nature of the organization where 
the implementation is taking place. For example, consider the frequent 
personnel turnover within the Navy, and the high probability that new 
rotations will not arrive with an expertise in OA use. This situation 
places a special burden on those whose job it is to insure continuity in 
work tasks. 

4. If effective plans are to be developed to maximize the con- 
tribution of information technology, the benefits of the technology can- 
not all be collapsed into a single metric. There are classes of outcomes 
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that need to be understood on their own terms. For example, two advan- 
tages of OA use might be: better graphs in briefing documents, and the 
accomplishment of more work due to time saved through computer use. 
Depending on which of these outcomes is most desired, different actions 
have to be taken in terms of training, demands made upon personnel, and 
rewards given for OA use. 
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4. MODEL FOR DATA COLLECTION 

We view the use of OA as an ongoing process that begins with im- 
plementation; has an immediate impact on end-users and their work groups; 
and through the activities of those work groups, spreads its impact from 
the immediate locus of use. (Work groups are defined as that small num- 
ber of individuals with whom a person regularly interacts in the pursuit 
of his or her assigned tasks.) Throughout the process, "feedback loops" 
are operating; that is, users of OA are influenced by the way their work 
affects others. This work- group/diffusion perspective lays the ground- 
work for addressing four important issues of practical concern to the 
Navy: 

1. What is the process by which groups adjust to OA, and why do 
some do it better than others? 

2. What is the most immediate impact of OA use? That impact is 
likely to occur through interaction between groups using OA and those 
who receive their products or services. Thus, a work group perspective 
provides a strong framework for assessing impact and justifying the ac- 
quisition of OA technology. 

3. How might the nature of work be affected by OA? Here too, the 
most immediate effects are likely to be at the work group level, where 
two types of changes might occur: (1) Work groups may become more adapt- 
able, ie. better able to handle a variety of tasks or respond to new 
circumstances. (2) Because of the ability to do different types of 
tasks, a work group's role within the organization may shift. 

In addition to its practical value, a work group/impact diffusion 
perspective offers important methodological advantages: (1) It allows 
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comparisons across many instances of implementation, thus yielding much 
useful information about the OA implementation process. (2) The pace of 
data collection can be matched to the actual pace of OA implementation. 
(3) There is a good starting point for assessing impact as it diffuses 
throughout the organization. Such a starting point is critical because 
as the diffusion of impact proceeds, it becomes progressively more dif- 
ficult to detect the role played by information technology in the general 
run of organizational processes. Sampling becomes more difficult because 
of uncertainties in knowing in advance who should supply data. The na- 
ture of OA's impact becomes harder to predict, thus making it difficult 
to anticipate specific questions which should be asked. 

Because of the above considerations, any OA evaluation plan must 
meet five conditions: 

• Implementation must be understood in terms of the work groups 
where that implementation is actually taking place. 

• The link between implementation and impact must be studied. 

• The impact of OA use must be traced as far as is possible, 
given constraints on time and access to data. 

• Interaction and feedback loops among elements in the 
implementation-use process must be included. 

• Because implementation and use are processes, efforts must be 
made to assess its evolution, and to track any important 
changes in direction which might have implic rations for 
planners . 

Figure 1 is a schematic diagram of a model which meets these 
conditions. 
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Beginning of 
implementation 



Implementation within 
work groups 



1 



Activity of 
work groups 

\\f — - 



Impact on immediate 
\ recipients of work 
groups products or 
services 



Impact on work group's 
s ability to do tasks or 
adapt to new 
circumstances 



nuation 



Diffusion of Impact throughout 
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necessary) 



through general 
acceptance 

This model will serve as a guide through four steps in the 
evaluation design process: 

1- formulation of a measurement strategy, 

2- identification of relevant variables, 

3- scheduling of data collection, and 

4- developing a sampling scheme. 



Fig. 1. Schematic illustration of analysis plan. 
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5. MEASUREMENT STRATEGY 

REVIEW OF PREVIOUS MEASUREMENT EFFORTS 

Others' efforts to assess the impact of OA provide a valuable con- 
text for conceptualizing any new assessment strategy. Those efforts can 
be conveniently grouped by means of a two-way table in which rows refer 
to OA technology, and columns refer to impact. In each case, the crucial 
distinction is whether variables are dealt with primarily in general or 
in specific terms. This framework is shown schematically in figure 2. 

Reference to impact 
QlQh&L Specific 

office automation 

Specific 



Figure 2 - Schematic representation of efforts to assess the impact of 
Office Automation. 

All of the cells in figure 2 are represented in the literature except 
for the "global -global w cell, which is so general that it is not used. 

The following are brief examples of how each category was dealt 
with in a variety of research studies. 
Global Technology - Specific Impact 

In an effort to determine the impact of OA on managers, Fleischer 
and Morell (1986, 1987) surveyed 168 managers in 15 organizations. One 
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part of their forced-choice nailed questionnaire asked respondents to 
use a five point scale to rate OA's impact on the amount of tine they 
spent doing 11 common business activities. Examples of elements of the 
list include: "talking on the telephone," "preparing presentations," 
"using data to make decisions," and "business travel." (The scale points 
were (1) much more time, (2) more time, (3) same time, <4) less time, and 
(5) much less time. A "don't know" category was also included.) 

Another example demonstrates how the "global technology-specific 
impact" approach can be used with a very different methodology. Bikson, 
Stasz and Mankin (1985) conducted a series of structured interviews in 
order to obtain an in-depth understanding of how computer-mediated work 
affected one large corporate headquarters. An important part of their 
data analysis was to determine how the technology affected a series of 
variables related to organizational behavior: (1) work changes (enrich- 
ment, demands, reinvention, management style), (2) communication, 
(3) ability to return to old ways of accomplishing tasks, (4) productiv- 
ity (time savings), (5) physical and psychological complaints, (6) for- 
mal job changes, and (7) job satisfaction. 

4 

Specific Technology - Global Tn, p a rf 

One example of this approach is provided by DeLong and Rockart 
(1986), whose effort to study executive support systems involved tele- 
phone interviews of knowledgeable information systems personnel in 45 
Fortune 500 companies. Part of their analysis involved imposing three 
categories of impact (extensive, moderate, and low), on three elements 
of executive support, - office automation, status access, and query and 
analysis. Each of these terms represented a step in an ascending scale 
of end-users' ability to access and manipulate information. 

ERIC > 0 
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A second example is again provided by Fleischer and Morell's (1986 , 
1987) study of managers. One section of their instrument was a long 
list of specific OA applications (word processing, spreadsheets, statis- 
tical analysis, etc.), *«ch of which had to be rated on a six point scale 
of impact on work life: (1) profound change for the better, (2) major 
beneficial impact, (3) some beneficial impact, (4) slight beneficial im- 
pact, (5) no impact, and (6) negative consequences have outweighed bene- 
fits. 

Finally, research of this type often substitutes "frequency of use" 
for direct estimates of impact. Laudon (1986) for instance, used this 
approach in his effort to assess the influence of networking on the use 
of personal computer (PC) technology. His study dealt with a variety of 
specific offices within 25 firms in the financial services industry. 
Data were collected by means of both open-ended interviews and through 
structured questionnaires filled out by the researchers. One aspect of 
the study asked respondents to estimate how often (never, monthly, week- 
ly, daily) they used PCs for a variety of purposes - searching data- 
bases, external communications, learning aids, and the like. 
Specific Technology - ggfiSlfjLs TlTimt 

One part of the Fleischer r.nd Morell (1986, 1987) study was an effort 
to ascertain how OA affects managers' decision making. A section of 
their instrument defined three types of problem: "cut and dried," 
"analytic," and "unstructured." These problem types form a decreasing 
scale of the extent to which specific information can help find a well- 
defined solution. Respondents were then asked to use a five-point scale 
to rate how helpful various aspects of OA were in finding solutions for 
each type of problem. 
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SOURCES OF DATA 
Archival Data 

Archival data has the great advantage of not requiring people to 
answer questions, thus eliminating a potential source of bias and mini- 
mizing requests for respondents' time. Although relatively few of the 
concepts detailed in this report are amenable to study through archival 
means, some are, and these should be used whenever possible. The few 
examples presented below will convey a sense of the type of information 
that might be available through such means: 

• The quality of written documents, in terms of format and use of 
graphics, can be compared before and after the advent of OA 
use. 

• People may be able to do more work as a result of time saved by 
the new technology. Thus one might look at the number of low 
priority tasks completed, or the number of products (plans, 
orders , budget* , etc . ) . 

If OA is used to generate information that was not previously 

available, that capacity should be reflected in the contents of 

plans and briefing documents. 

Unfortunately, much of the data needed will not be available without 
asking people to take the time to answer questions. Because time is a 
scarce resource, a strategy is needed to make personal data collection 
as efficient as possible. 

An important element of that strategy is to collect as much infor- 
mation as possible during the normal course of implementation and system 
management. Because those activities require frequent contact with 
end-users, much evaluation data can be collected in this way. For ex- 
ample, much information on training needs could be garnered from ordin- 
ary discussions between help center consultants and end-users. A little 
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foresight would insure that training data were collected in a systematic 
fashion that would allow its incorporation into a formal evaluation. 
Multiple Sources of Data 

Multiple perspectives are always useful as a way of checking the 
validity of data. For example, we have encountered many users of OA who 
claimed that computer use resulted in their being more convincing in 
their arguments. The better arguments came from better use of graphics, 
more frequent revisions of documents, and the inclusion of more relevant 
data. Examples and details provided by respondents indicate that their 
claims are probably true. But how is the situation seen from the point 
of view of people on the receiving end of those arguments? Is the in- 
crease observable to them? Is it large enough to make a practical dif- 
ference? 

A second perspective on multiple data sources is the combination 
of personal and archival data. To continue the example cited above, 
more convincing arguments might be reflected in the use of graphics in 
briefing documents, or a greater richness of data used to support argu- 
ments . 

Another advantage of multiple data sources is that they can provide 
a sense of the "size" of any change that occurs. In our "better argu- 
ment" example, OA users may be able to document that new information is 
included in their reports, even though an increase in quality is not per- 
ceived by the recipients of those reports. In such a case, we would 
have a sense that OA has had an impact on quality, but that some action 
may be needed to increase that impact co the point where it makes a prac- 
tical difference. 
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In sum, any use of multiple data will promote greater confidence in 
findings, a higher probability of detecting events, and a finer sense of 
how consequential any detected change may be. 



ASKING QUESTIONS 
Question Construction 

In all cases where data must be collected from individuals, the 
goal is to work towards a closed ended question which can be answered 
easily, allows comparisons across settings, and which is anchored in the 
reality of the respondents' work. For example, respondents may be asked 
if they use their OA to keep "private" files to make up for deficiencies 
in a mainframe system. This is a very specific question which conveys 
information about one special use of information technology. On the 
other hand, the question is general enough that it applies to many con- 
texts, and can be aggregated across diverse respondents. 

The only way to construct relevant questions is to begin with a 
small number of open-ended interviews, and use that data to generate spe- 
cific response categories. Two examples from a study done for the Naval 
Military Personnel Command illustrate this process, (Morell 1987). 

Context of the study: Individual work groups obtained 
micro -computers on their own initiative and taught themselves how 
to use the technology. Witnout specific orders to do so, they 
began to use computers to work more efficiently. Although 
micro- computer use greatly affected the work groups' operating 
routine, that use was not embedded in a careful "microcomputer use 
support plan." 

Example # 1: Given these circumstances, we became concerned with 
difficulties that might arise when experienced users rotated out of 
their jobs and new people came in who were neither committed to the 
technology, nor expert in its use. To address this question we 
asked i a open-ended question about what would happen when the end- 
user rotated out of his or her position. Although all answers were 
slightly different, it was possible to collapse them into four 
basic categories. 
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• Routines are institutionalized by setting up data entry 
screens, computerizing files, and the like. The assumption 
here is that new personnel cannot ignore so institutionalized a 
system. 

• Replacements are sought who already have some facility with OA. 

• OA-related duties are transferred to a person who is remaining 
in the group. 

• Use is left to the discretion of the new person. 
Following this example, one part of a question about "strategies 

for continuity" might be to list specific strategies, and ask respondents 
to estimate their groups' reliance on each. Then, as a check on these 
categories, respondents would be asked to briefly describe what was act- 
ually done when an end-user rotated out of his or her position. That 
description would be used to further refine the categories in the forced 
choice part of the question. As this process proceeded, it would become 
less and less necessary to ask people to write or narrate long answers. 

Example #2: One set of open-ended questions in the study was 
designed to ascertain the impact of personal computers on 
activities related to planning and administration. Analysis showed 
that several themes were common to a wide variety of responses. 
First, many users talked about uses where the primary advantage was 
getting information faster. A second theme was the advantage of 
obtaining more current information than would otherwise be 
available. Third was the notion of a wider range of data. 
Finally, there was a newfound ability to interrelate sets of data 
elements. Although there was a great diversity in what people 
actually did with their machines, these four themes were able to 
summarize almost all of the responses. 

There are several advantages to abstracting themes as was done 
above. First, doing so removes the unit of analysis from the unique use 
of any given end-user. Thus, it becomes possible to aggregate informa- 
tion across contexts, or to compare settings. Second, these themes are 
close enough to the experience of end-users (and those they deal with) 
that proper questioning technique still has the potential to elicit 
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meaningful data. Finally, there may well be opportunities to measure 
these factors through archival means. As examples, one might find a 
paper trail relating to the adequacy of budget projections, or to the 
number of requests for ad hoc data analysis. 

Advice about using short questions will become progressively more 
difficult to follow as impact is traced further and further from the 
immediate source of u:.*. This is because as impact diffuses, there is 
less prior knowledge about who will experience that impact, or in what 
ways. What is certain is that diffusion of impact will bring ever great- 
er difficulties in teasing out the effect of OA from the multitude of 
events that govern organizational process. Thus, a commitment to a broad 
understanding of impact must be accompanied by a willingness to engage 
in open-ended interviewing. To uncover the sought after information, 
that interviewing must be based on a deep prior knowledge of how OA has 
been used. 

Question format. Should data be collected in questionnaire or in- 
terview format, or some combination of both? In the early stages, inter- 
viewing is required to make sure questions are appropriate. Once ques- 
tions are refined however, most of the data can be collected in either 
an interview or a questionnaire format. The choice must strike a balance 
among such factors as respondents' reaction to different formats, re- 
searcher time, and finances. 

Beyond the issue of format for individual questions, the matter of 
formatting sets of questions in an interview or questionnaire must be 
considered. In general, the best policy is to begin with factual ques- 
tions that are easy to answer, and will facilitate respondents' answering 
more subtle questions. As an example, consider the following 

o 31 
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possibilities for ordering of sets of questions about the impact of OA 

technology on work life. 

Order 1 Order 2 

a- description of OA technology ways the use of OA has changed 
available to the the tasks done 

respondent; by the respondent 

b- amount of time respondent amount of time respondent 

spends using various spends using various 

elements of OA; elements of OA 

c- ways the use of OA has changed value judgments about 
the tasks done the impact of changes 

by respondents; due to OA 

d- value judgments about description of OA technology 

the impact of changes available to the 

due to uses of OA. respondent 

Order 1 follows a logical progression. Each set of questions is 
more "concrete" than the set that follows it. Also, each set calls for 
information that is prerequisite for answering the set that follows. By 
the time respondents get to the "value judgments" section, they will 
have refreshed their memories as to how they have used OA and how it has 
changed their work life, and thus be able to think seriously about the 
difficult questions that make up the last section. Order 2, however, 
does not follow such a progression, and is thus much less likely to elic- 
it the desired information. 

MAGNITUDE OF IMPACT 

The previous discussion of multiple measures raised the notion that 
any given impact of information technology can vary on a continuum from 
"strong" to "weak". This concept must be applied to all important vari- 
ables if an evaluation of OA use is to be truly useful. One reason for 
its inclusion is to convey a sense of where action is needed in order to 
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improve the value of computer use. Another reason is that it allows 
judgments about the value of the technology to the Navy. 

Consider the use of OA to aid with budgeting for small "shops" with- 
in the Navy, which we have found to be a common application. What might 
be some of the consequences of this budge? keeping? 

• People may gain a newfound ability to finish their work on 
time. 

• There may be time to accomplish low-priority tasks which were 
previously uncompleted, or not done well. 

• Better accounting may lead to fewer funds being "lost" by the 
group to the system. 

• Knowledge of when funds are available aiay result in better 
service by the group to the Navy, as for example, when a 
detailing group needs to cancel or reschedule fewer orders for 
lack of funds. 

Although some value judgments are involved, most people would agree 
with two assertions: 

• These outcomes are generally listed in ascending order of 
importance to the Navy. 

• Each outcome has some value in its own right. 

If there were available data on only one of the outcomes, there would be 
little sense of what kind of a contribution OA was making, or the extent 
to which it was fulfilling its potential. 

VALUE JUDGMENTS ABOUT CHANGE 

Because value judgments about the importance of observed changes 
are inescapable, data on those judgments should be formally incorporated 
into any assessment strategy. Consider the "order rescheduling" example 
in the previous section. If the orders involve assignments of scarce 
personnel to critical specialties, a small increase in efficiency could 
have very important benefits. When dealing with billets that have large 
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numbers of qualified people, and which do not require highly specialized 
training, small increases in detailing efficiency may be a minor con- 
tribution, at best. 

A small group of people charged with evaluating OA cannot presume 
to make these value judgments for all the diverse settings they will be 
studying. Thus in addition to questions about what actually happened, 
it is critical to ask respondents about the importance of those events. 

MEASURING PRODUCTIVITY 

Because productivity is such an important issue in the evaluation of 
OA, it is worthwhile to clearly define a strategy for determining how 
productivity should be measured. In addition to the general issues of 
measurement mentioned above, there are several special issues that must 
be considered when measuring productivity. 

The first step is to catalogue all accepted measures of a work 
group's productivity. OA may not affect all of those measures, but com- 
paring specific impact with the full range will provide a sense of the 
extent to which OA affects the group's functioning. Furthermore, data 
based on secepted indicators are more likely to be understood and used 
than will be findings based on unfamiliar measures. 

The next step is to elicit information about the appropriateness of 
those measures. Suppose, for example, a group is rated equally for ac- 
complishing high and low priority tasks. This might be defensible for 
some purposes, but it can lead to a distorted picture of the impact of 
OA technology. 

It is also possible that accepted measures do not capture the 
essence of a group's contribution to Navy functioning. For example, a 
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group may be rated on whether it develops plans in time to be incorpor- 
ated into a general planning process, but is not rated on the plan's 
contribution to that process. Such a possibility is quite likely because 
on- time completion is easily observable, while quality is not. Again, 
this might be acceptable from a bureaucratic point ot view, but reliance 
on the "on- time" measure would indicate that OA is having a less impor- 
tant affect than it actually is. Thus, although accepted measures of 
productivity can serve as a starting point, they should not be assumed 
to be the only relevant indicators. 

In particular, it is important to be sensitive to the "value added" 
elements of productivity that can be easily overlooked when using stan- 
dard organizational measures of productivity. One aspect of "value add- 
ing" is the ability to perform new, or different types of tasks which 
are not easily observed in their own right, but which may improve the 
quality of a product or a service. A second type of added value is "in- 
formation creation," a concept which can be viewed as analogous to "job 
creation" within the economy. Just as economic activity can generate 
employment opportunities that did not previously exist, use of a computer 
might generate new information which has value for the organization. 

Information must also be collected on the difficulty of effecting 
observed changes in productivity. For example, an order processing group 
may - with a little extra effort but without the use of fancy technology 
- easily increase its productivity by 15 percent. But a group charged 
with in-depth analyses of complex problems may find it very difficult to 
increase their output even slightly. For them, and for those who rely 
on their work, a small change might require an OA system, and might be a 
very significant contribution to Navy functioning. 

ERIC 
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The purpose of this section was to articulate a methodology for 
developing measures related to the implementation and impact of OA sys- 
tems. The intent was to show has things should be measured. We now 
turn to a detailed list of what should be measured. 
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6. DATA TO BE COLLECTED 



The presentation in this section will follow the order represented 
in the model presented in the previous section: implementation, activity 
within work groups, uses of the technology, immediate impact of work 
group outputs, changes in the role and adaptability of work groups, and 
remote impact. 



IMPLEMENTATION 

The variables proposed to study implementation are drawn from the 
social science literature on innovation adoption, and modified by our 
knowledge of salient issues within the Navy: 

!• Reason for adoption Innovation implementation tends to succeed when 
it is nggi flrlygn rather than technology driven . Thus, OA use will 
develop more quickly within groups who begin with a clear sense of 
the need for such technology. Those who obtain OA simply because it 
is available, will tend not to make good use of their acquisitions. 

2. Kev players, critical mass Within each setting, how many people are 
active advocates for the use of OA technology? OA use should increase 
with the size of this group, because increased size: (1) increases the 
probability of a person quickly receiving technical and moral support, 
and (2) makes it easier to induce change in the work style of a group. 

3 - SWPOrt for training and OA use Effective use of a new technology 
requires that users' have both the opportunity to learn and the 
ability to make whatever changes in work-style are needed to exploit 
the technology. This support can be manifested in different ways - 
support from a superior, assistance from colleagues, or discretion 
over the use of one's time. 

Available expertise Effective OA use requires some combination of 
expertise by end-users, and available help when problems arise. As a 
result, it is important to assess training opportunities, previous 
knowledge of OA, and extent of available expert help. 

5 - Adaptive planning/user participation Do the users have input into 
how their OA will be configured and used? The greater the user 
participation, the greater will be the extent of OA use. 

6 - Incentives for use Perceived advantages must outweigh the 
difficulties of: (1) learning to use the technology, (2) adapting 
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work style to exploit the technology, (3) time spent in data entry, 
and (4) any other difficulties that users may perceive. 

Another important aspect of implementation is the extent to which an 

innovation is routinized within an organization. A routinized innovation 

stays after its original supporters have left and people's attention has 

turned to other priorities. Because of the staff turnover problem in 

the Navy, the goal of insuring rout inizat ion is particularly important. 

In order to understand how different groups handle the routinization 

problem, several specific issues must be studied, (Unfortunately, no 

real examples could be found of instruments which used these variables 

to study information technology). 

• formal training for new personnel; 

• informal arrangements to acquaint new people with automated 
procedures ; 

• efforts to insure that incoming personnel have a familiarity with 
OA f 

• construction of software routines to make OA use easy for the 
uninitiated; 

• organizing work to require OA use; 

• setting expectations that OA will be used; 

• recognition for people who make good use of OA; 

• transfer of OA-related duties to personnel who will be at their 
jobs for the longest possible period of time; and, 

• transfer of OA duties from a person who is leaving to one who is 
remaining. 

WORK GROUPS 

In order to plan effective implementation, several types of informa- 
tion about work groups are needed. 
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Basic Description of Work Groups 

It is useful to have a sense of how the work groups fit into their 
larger organization, of the internal structure of the groups, and of the 
work done by each group - whether or not that work relates to information 
technology. This contextual knowledge will help in understanding how 
much of a group's work can potentially be affected by the new technology, 
how that work might contribute to the organization, how large a change 
in work routine has been brought about, and the possibilities for further 
change , 

OA agfti A second type of knowledge concerns how. the work group 
actually uses its OA on a routine basis. One facet of this knowledge is 
the applications used by members of the group. Another facet is the 
role played by OA in carrying out assigned tasks. 

Satisfaction with OA technology- Over and above knowledge oZ what 
the group does, it is extremely useful to know how satisfied end- users 
are with their OA, Such information is valuable in understanding why 
the group structured its OA related activities as it did, and what limi- 
tations need to be overcome if the technology is to be used for other 
purposes. 
Impact of OA 

A full understanding of the impact of OA technology must include 
data on the technology's impact in six areas: 

• amount of products or services produced by user groups, 

• quality of those services, 

• groups 9 ability to adapt to new circumstances, 

• changes in the group's role in the organization, 
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• communication patterns, and 

• larger (outside the work group) organizational consequences. 
Data collection on these topics must be based on principles set 

forward in the section on Measurement Strategy • multiple measures, mul- 
tiple perspectives, personal and archival data, remote impact, and as- 
sessment of both the amount and value of change. Schematically, such a 
data collection effort can be depicted as shown in figure 3. 

Amount of production. Here we refer to the " things" that a work 
group actually transmits outside of its boundaries. Examples include 
advice, reports, orders, budgets, briefings, and plans.. The important 
issue is not how much it produces, but how much it produces relative to 
the personnel on hand, i.e., the group 9 s productivity. 

As an aid to understanding their impact, it is useful to classify 
these outputs in two ways: 

• more outputs of the type always produced, versus new types of 
outputs; and 

• budget or financial related work, versus non-financial aspects 
of planning and administration. 

These distinctions provide a sense of what elements of work are 
being affected by OA, and of how the nature of that work is changing. 

Qvallty. In addition to the amount of output, OA can also bring 
about a change in the quality of production. Consider the following 
examples: more precise schedules; more accurate forecasts; personnel 
assignments better matched to the needs of a job; quicker action on re- 
quests for information; and better data for analyses. 

Adaptability, Because of its members' facility with information 
technology, a work group may increase its rapacity to adjust activities 
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Figure 3 - structure for collection of impact data 



to new demands or circumstances. Since adaptability is a critical factor 
in organizational success, it is important to understand how OA contri- 
bute to adaptability, and thus, how that contribution can be maximized. 
Examples of increased adaptability include speed of responses to ad-hoc 
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requests for information; the format of responses (use of graphics, elec- 
tronic communication, etc.); the shape of the "learning curve" for new 
tasks, the capacity to obtain and analyze new types of information, and 
the capacity to produce different products or services. 

Changes in work group's ac tivities . As recognition spreads about 
what a work group can do because of its use of OA, the group may begin 
to experience new demands made upon it. Examples of such demands might 
include requests for different information or analysis, or giving the 
group oversight of new activities. In some cases these new demands might 
involve a formal change in the group's mission. More likely, formal 
roles will remain intact, but new demands will be made as people recog- 
nize those groups' new capacitv, and follow a natural inclination to use 
all available resources to get a job done. 

Knowledge of evolving roles for work groups is important for both 
short and lcng term decisions. In the short term, managers need to know 
what resources are available for accomplishing tasks. In the longer 
run, a sense of the direction of evolution in an organization is impor- 
tant when, as inevitably happens, choices have to be made about formal 
changes in structure and assignments. 

Communication. A large part of how a work group interacts with its 
environment can be understood in terms of communication patterns between 
the group and the rest of the organization. Three elements of communica- 
tion are important - the parties involved, the form of communication, 
and its content. Through these one can discern a work group's importance 
in the scheme of things, the way it contributes to organizational func- 
tioning, and the impact of its contributions. Because this topic is so 
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important, it should be treated separately, and not lost as an after- 
thought in the general category of "work group functioning." 

Organiza tion-vide change . Although the most immediate and dramatic 
impacts of information technology are closely tied to work groups, one 
should not lose sight of other important organizational changes. For 
example, better budget keeping throughout many parts of an organization 
can have fiscal implications far beyond any small group's ability to 
keep better track of its money. A complete study of the impact of infor- 
mation technology must include a determined effort to assess these larger 
scale changes. 
Ne gative Consequences 

Although we have found the impact of OA to be generally beneficial, 
the possibility of negative consequences cannot be ignored. Any outcome 
- quantity, quality, adaptability or group role - can be adversely af- 
fected. Thus in the course of data collection, it is important to ask 
questions and search for information in ways that will detect both posi- 
tive and negative consequences. 
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7. THE DATA COLLECTION PROCESS 

TIMING OF DATA COLLECTION 

Two facts must be borne in mind when setting up a schedule of data 
collection. First, comparisons of impact before and after implementation 
are importar.it. Such comparisons make it possible to assess the affects 
o£ particular implementation efforts, and to estimate the amount of 
change wrought by OA. Second, some data in the evaluation will be much 
more time-sensitive than others. While some data can be collected at 
any convenient time (archival information, for instance), others must be 
obtained within a narrow window of opportunity. 

One type of timing problem is waiting too long; and delay in data 
collection is a major timing problem, as accurate recollections of spe- 
cific pre- implementation circumstances are lost. For example, consider 
how one can judge the quality of recommendations made, based on an analy- 
sis of the type, amount and currency of information that went into those 
recommendations. In some cases, those recommendations will be contained 
in short memos or briefing documents which do not reflect the depth of 
analysis that went into their making the recommendation. If people are 
not interviewed prior to implementation (or very sh rtly thereafter) , it 
may be impossible to make any kind of before -and- after comparison. 

A second timing error is collecting data too soon. Any innovation 
is accompanied by an adjustment period during which productivity may 
well decrease. If people must be interviewed, it may be necessary to 
time data collection for as far past the actual implementation as pos- 
sible, while still reaching people before they rotate out of their 
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positions, or before their accurate recollection of the pre -implementa- 
tion setting fades. 

SAMPLING STRATEGY 

In the early stages of implementation it's important to collect 
data from as many end-users as possible, and from as many people as pos- 
sible who are affected by end-users. Important sources of data include 
members of end-users 1 work groups, their commanding officers, and recip- 
ients of the products of information technology. Extensive data collec- 
tion will be required on this small sample because of the important con- 
tribution they will make to the instrument development process. 

As usage spreads, however, the number of potential respondents will 
quickly exceed anyone's capacity to collect data. At that point it will 
be critical to develop a sampling scheme. Although the precise nature 
of that scheme cannot be specified in advance, it is possible to articu- 
late basic principles to guide the process. 

Any groups of particular interest to those charged with OA implemen- 
tation should be sampled. These may include groups where data is needed 
to help solve an implementation problem, or groups which are succeeding, 
and thus deserve special mention. 

The remainder of the sample must be representative of the range of 
people affected by OA use. This includes end-users, other members of 
their groups, and users of OA based information. 

Because some data must be collected early in the implementation 
process, groups and individuals targeted for inclusion in the study 
should be identified as early as possible. 
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There are some groups who should be included in the study because 
of their special relationship to the OA use process. For example, "sup- 
port for use" and "available expertise" are important in the implementa- 
tion process. Thus, people involved in end-user support and 
training may have valuable insights that should be included in ^fce 
analysis . 

OA users are likely to have a variety of customers for their pro- 
ducts or services. In order to assess the full range of OA's impact, 
this variety should be reflected in any sample of those customers from 
whom data is collected. A corollary of this principle is that a prior 
analysis must be done in order to determine who those customers are. 

The final guideline about sampling deals with what data shoulJ be 
collected from those who are selected. It should be obvious by now that 
a very great deal of information will need to be collected, and that 
there is a great risk of placing too heavy a burden on any given respon- 
dent. One solution is not to ask all questions of all respondents. 

As an example, one might want to address many questions to the re- 
cipients of reports that are generated with the aid of OA. Included in 
the list would be questions about: timeliness, readability, format, 
length, number, revisions, relevance, information drawn upon, and quality 
of analysis. Although each of these items is important, it may not be 
important to ask each recipient of each report to address every issue. 
A few questions may be so important that they should always be included. 
Others, however, might be addressed to only a subset of recipients, or 
the entire question set may be subdivided. Many variations on this theme 
are possible, all of which require a trade-off between the completeness 
of the data set and the burden placed on respondents. 
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8. EXTENDING THE INNOVATION TO NETWORKING 

Networking and stand-alons workstations must be considered as sep- 
arate innovations. Although the basic approach to their evaluation is 
the same, there are important differences in emphasis. For example, 
understanding computer- computer interaction may require an analysis of a 
setting's mainframe environment, or of group processes related to agree- 
ments over fi?we structures and data elements among linked workstations. 
Issues such as these are not critical for understanding the use of 
stand-alone workstations. These differences of emphasis apply to studies 
of both the implementation and impact of OA technology. 

IMPLEMENTATION 

In terms * T implementation, networking will require a much greater 
attention to the role of key players, and to the use of an adaptive 
planning strategy. The reason for this is that networking has the effect 
of bringing about tighter linkages among parts of an organization, thus 
increasing the mutual influence of innovation on organizational process, 
wid vice versa. As examples of this process, consider the following 
questions, each of which indicates how networking might act to tighten 
organizational linkages. 

• What might happen to relationships among groups who previously 
had unique functions, but whi^h relied on each other for infor- 
mation? 

• Would the extent of cooperation and mutual dependance change 
among groups with mutual interests, but who did not previously 
interact? 

• Could networking bring about an integration of functioning 
across previously autonomous groups? 

• What are the implications for the overseers of mainframe sys- 
tems when many users have the capacity to access data? Are 
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there legitimate concerns about input/output capacity, file 
integrity, and security? 

• How much power will groups have to give up when they begin to 
share their information with others? 

• Once a group has gone to the trouble of setting up its own 
"private" databases, will it readily agree to participate in 
a new shared file system that will require changes in their 
already proven file structure? 

As organizational linkages tighten, those in charge of implementa- 
tion must work carefully to bring about effective coalitions among groups 
with diverse interests. Doing so will require a careful planning process 
that recognizes the legitimacy of divergent needs, and which can facili- 
tate compromises among them. The success or failure of that planning 
effort will have a profound affect on the ultimate shape of the innova- 
tion. 

While the importance of key players and adaptive planning will in- 
crease as implementation moves from stand-alone machines to networked 
systems, there may be a corresponding decrease in the importance of other 
implementation variables - support for training, available expertise, 
and similar factors which relate to the ability of an individual end-user 
to operate the technology. The reason for a decreased emphasis on these 
factors is that by the time networking goes into effect, the general 
level of computer literacy among users may have risen quite high because 
of their previous experience with the stand-alone systems. 

IMPACT 

The shift from stand-alone to networked systems also has implica- 
tions for evaluating outcome. Although the variables of interest 
remain the same (output, quality, adaptability and evolving roles), there 
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are shifts in the likelihood that changes will occur, and in the direc- 
tion , and size of those changes. 

There is an increased risk that networked systems will have no im- 
pact, because changes can only occur if a system is implemented, and as 
we have seen, organizational difficulties increase the chance that ef- 
fective implementation will not happen. 

Linking workstations increases the dependence of different parts of 
the organization on each other. As a result networked systems are more 
likely than stand-alone systems to produce undesirable effects. This is 
so for three reasons. First, technical malfunctions will affect all 
groups linked to the system. Second, the more sharing of information 
that is done, the greater the negative consequences of sharing incorrect 
information, or of disseminating error-prone programs. Third, sharing 
increases the dependence of work- groups on each other, thus increasing 
the difficulties if one of the work groups does not do its job well. 

Finally, networked systems have the potential to result in larger 
changes than may be observed with free standing workstations. This is 
because networking may free many work groups from the tedious job of 
data entry, thus providing opportunity to accomplish other work. More 
important, networking provides a vast increase in available information 
and much faster communication. 
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PURPOSE AND STRUCTURE 



This Appendix provides specific examples of questions chat can be 
used to assess the implementation and impact of office automation (OA). 
As much as possible, multiple examples are presented from a variety of 
studies that have actually been carried out. This range of questions is 
intended to help the reader in three ways. 

1. The large number of examples may include a question that can be 
used as is, thus saving effort in the item construction process. 

2. Showing how similar information can be sought in different ways 
will help the reader adapt questions to special circumstances, or 
to write new questions as the need arises. The ability to 
construct questions is important because studies of OA often have 
unique needs for information which cannot be addressed by an 
existing stock of questions. 

3. In some cases we present a series of questions on the same topic 
from the same study. Inspection of these series will convey a 
sense of how multiple questions can be used to get at different 
aspects of an issue. 



Much of *he wording in these questions is not appropriate for a 
military context, as these studies were conducted in a variety of 
settings. The original wording is presented here in order to remain as 
faithful as possible to how the authors' actually constructed their 
questions, and to convey a sense of how these researchers went about 
their work. 



The examples are arranged in sections which correspond to the 
variables mentioned in the text of the report. This arrangement 
facilitates finding questions on any given topic. What is lost, 
however, is an overall sens* of how a questionnaire should be put 
together. 
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INSTRUMENT CONSTRUCTION - BRIEF REVIEW OF PRINCIPLES 



Writing questionnaires and interviews is an art in which the author 
must guess what order of questions will best fulfill several criteria. 
First, respondents' interest must be attracted and held throughout the 
series of questions. It would probably not do, for example, to begin 
with a boring set of detailed questions about the minutia of a 
respondent's education. 

Second, the order of questions must be chosen to help respondents 9 
locate the information they are seeking. As an example, consider an 
opening question such as: How has the computer affected the 
productivity of your work group? This can be an almost impossible 
question in the absence of previous questions which helped he respondent 
recall information needed to make an informed judgement about 
productivity. Better to begin with questions about how the computer is 
actually used, and under what circumstances. The answers to those 
questions are close to the respondent's immediate knowledge, and the act 
of answering forces a review of Information that will allow a thoughtful 
response about the more difficult issue of assessing productivity. 

Third, the appearance of sensitive questions may make respondents' 
unwilling to continue. Clearly it is best to leave such questions 
toward the end. 

The above three considerations lead to a general prescription for 
questionnaire construction. Bagin with questions whose answers are both 
interesting and easily available to the respondent Use questions early 
on to help people remember information they will net»d to address 
difficult questions. End with questions that are likely to be sensitive 
or boring. 

It is impossible to always follow the above advice, as much of it 
is contradictory. As examples, questions that are easy to answer might 
be boring, or sensitive questions might have to appear early to help 
respondents make difficult judgements. There is no formula. There is 
only general advice which must be tempered by a knowledge of a 
particular situation, and bv as much pretes ting as possible . 

Finally, no matter how appropriate an ordering of questions, no 
interview or questionnaire will work if questions are not clear. The 
intent of the question, as well as its vocabulary, must be understood. 
Here too, pre-testing is of immense help. 

A critical decision is the choice between using an interview and a 
questionnaire format. Interviews give the researcher two valuable 
opportunities. First, unanticipated - but interesting - issues can be 
pursued in depth. Second, the researcher can rephrase questiens or 
change their order, and thus avoid many problems of sensitivity and 
miscommunication without recourse to elaborate pretesting. (No 
interview will go well, however, unless the interviewer has a clear 
sense of what questions will be asked, why those questions are 



50 



important, and hov respondents nay react to particular questions.) 
Related to the smaller pretesting requirement is the advantage that 
instruments can be developed relatively quickly. 

Unfortunately a high price is paid for the flexibility of open 
ended questioning. Part of the price is that by requiring the presence 
of a researcher, open ended questions severely limit the amount of data 
that can be collected. Another problem is that the length and 
unstructured format of open ended data make analysis very difficult. 
The time saved in instrument construction can easily be gained back in 
the analysis process. 

The above discussion contains an implicit assumptior that is 
usually correct. Open-ended questions tend to be administered 
personally, while short answer and forced choice questions tesid to be 
part of wriU*n questionnaires. This need not, hov; 9 - always be the 
case. As ta example, consider a situation where th • „fc of the 
respondent essentially requires that questions be ao**nistered 
personally. The actual instrument may still contain manv forced choice 
and short answer questions, even thougt, the respondent answers these 
questions within the context of an interactive personal interview. 
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OVERVIEW OF QUESTION SOURCES 



There are relatively few empirical studies of OA, and only a small 
subset of those publicise their data collection instruments. Thus the 
pool of sample questions for this appendix was small. The questions 
presented here derive from literature reviews we have done over the 
years, or from studies we have been involved with personally. Below are 
short summaries of those studies. These summaries are intended to 
provide a sense of both the context in which the questions were asked, 
and the range of organizations where this type of research has been 
carried out. 

CQOTmter Mediated Work - Individual and Organi sational Impact In 
One Corporate Headquarter* * Tora K. Bikson, Cathleen Stasz, and Donald 
Mankin, Santa Monica Cal., The Rand Corporation, 1985, (Prepared for the 
Office of Technology Assessment, U.S. Congress, document #R-3308-OTA) . 

"This report describes how computer-based inforuation technology 
was introduced into one white-collar work setting, and explores the 
consequences to employees and the organization. The research extends 
prior work on information systems in varied user contexts and 
illustrates factors that underlie successful technological innovation in 
organizations . " (From the Preface, page iii). 

This research used work-groups as the unit of analysis, and 

semi-structured interviews as the means of data collection. The number 

of respondents, and their place within the organization, are listed 
below. 

"Organization 
Executive management (2) 
Personnel department (1) 
Technical department (2) 

Key actors, that is, other people who played key roles at 
the organization level in the implementation process (2) 

Work Group 

Department Heads (4) 

Linking actors, that is, individuals outside the focal work groups 
identified as links in the intra- organizational diffusion 
process (8) 

Individual: employees of focal departments (20) " (p7). 

An important element in this study was its attempt to collect data 
on the same issues from the point of view of different people in the 
organization. This is clearly shown in the way the same questions are 
asked in slightly different ways to a variety of respondents. Whenever 
the form of ths question changes in important ways depending on the 
respondent, these alternative versions are presented. 
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Personal and Org anizational Con sequences of Microcomputer use in 
the Naval Military Personnel Coimnand Jonathan A. Morell, Oak Ridge 
National Laboratory, Oak Ridge TN. 1987 (ORNL-6370). 

This study was an effort to assess the impact of personal computer 
use (PC) within the N4 (detailing) branch of the Naval Military 
Personnel Command (NMPC). The primary goal was to obtain a sense of how 
PCs were being used within NMPC; and the impact of that use both on 
people's work lives and on the organization as a whole. A second 
objective was to obtain the knowledge required to facilitate wider scale 
implementation of PCs, and to assess the impact of that implementation. 

The study employed semi -structured interviews to collect in-depth 
information. About six months after the initial interview, efforts were 
made to re- interview the respondents in order to see what changes had 
taken place in PC use. 

Wang Word Processing Sy stems: A Descriptive Study of Users and 
Quality of Training Programs Norma Rotman, unpublished Masters thesis, 
Hahnemann University, Program in Evaluation and Applied Social Research, 
Philadelphia PA, 1985. 

"In order to expand the existing limited knowledge about 
individuals utilizing word processing in their offices, a study was 
conducted on users of WANG systems in New Jersey, Delaware, and 
Pennsylvania. Over 600 self -administered questionnaires were completed 
in this non- randomized, one-group post-test research design. 

Training issues, white collar productivity, human factors, 
ergonomics, and user and company profiles were analyzed with SAS using 
cross tabulations, frequencies, and chi squares." 

Office Automation Rese arch Project: Manager Questionnaire 
Preliminary results from this study can be found in: Mitchell Fleischer 
and Jonathan A. Morell (1) : Managers as Informa tion Technology End Users 
proceedings of the 1986 International Congress of Technology and 
Technology Exchange, Pittsburgh PA, Oct 6 - 8, 1986. Further details of 
the study are available from the authors. 

This study was intended to shed light on two issues, first, how 
microcomputer use affects managerial work, and second, factors which 
influence the extent of managers' use of OA. Data from a closed-ended 
questionnaire was collected from 168 managers in fifteen large 
corporations. Major sections in the questionnaire were: (1) use of 
office automation applications, (2) process of equipment acquisition, 
(3) OA's fit with respondent's work life, (4) changes in activities due 
to OA, (5) technical support for use of OA, (6) quality, availability 
and use of training, and (7) characteristics of the respondent and his 
or her company. 
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Ag??ggment of an Effort to Integrat e Computer Functions in an 
Engineering Design Vi m Jonathan A* Morell and James Leemon Data Base 
vol- 18, #2, 1987, pp. 17-21. 

This study is an assessment of a large engineering design firm's 
efforts to integrate diverse elements of computer use. Three basic 
issues were studied: differential impacts on professional and clerical 
workers, impact of the system on document production, and effects the 
system had on the nature of work in the organization. Open-ended 
interviews were used to generate a closed-ended questionnaire. Data 
were collected from 28 technical personnel and 25 clerical workers. 

General Survey of Office Automation Tig** Results reported in: 
Managers as Information Technology End Users, Mitchell Fleischer and 
Jonathan A. Morell (2) proceedings of the 1986 International Congress on 
Technology and Technology Exchange, Pittsburgh PA, Oct 6-8, 1986. 

This was a survey of 88 working managers who were either enrolled 
in executive MBA programs, were part-time MBA students, or who were 
part-time students in a graduate program of Public Administration. Most 
questions required a short written response to a specific open-ended 
question. Respondents were asked about their personal use of OA, OA use 
by their immediate superior, and such use by a person supervised by the 
respondent . 
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EXAMPLES OF MEASUREMENT 



As much as possible, examples in this section are taken as-is from 
their sources. An attempt has been made to preserve the original 
layout, in order to provide a sense of how questions like these should 
be presented. Some liberties have been taken, however in order to make 
the layout conform to the format of this report. When a series of 
questions are presented from the same study on the same topic, an 
attempt was made to preserve the order in the original source. This was 
not always possible, however, because of the attempt to present 
questions within the categories of variables used in the body of this 
report. (Those categories do not always perfectly coincide with how 
other researchers structured their efforts). 

In some cases a question might legitimately appear as representing 
two variables. Good examples appear in the categories of " training" and 
"available expertise", where there is a fine line between the functions 
of training and consultation. Other difficulties of categorization 
appear when items within the same forced choice question seem better 
placed in multiple categories. An example appears in a question on the 
impact of micro -computers, where respondents are asked to rank both the 
impact of micro -computers on "coordination among groups" and "dollar 
savings." The first touches on work life, while the second is a value 
judgement about outputs. In order to convey a sense of the original 
question, the items are not split into separate categories. When either 
of the above two ambiguities appear, questions are placed in the 
category that represents their major emphasis, even if it is not its 
sole emphasis. 
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REASON FOR ADOPTION 



SP^rgg; — Bikson et al.. Protocol for Te chnical Expert 

1. What were the organization's main objectives in converting to online 
computer use? (If productivity, efficiency or effectiveness is 
mentioned, probe for how it is defined or measured.) 

2. How do the missions of the focal departments fit into this overall 
picture? 

Source: Bikson et al.. Protoc ol for Managers 

3. What were you trying to accomplish in converting to online computer 
use? (If productivity, efficiency or effectiveness is mentioned, 
probe for how it is defined or measured.) 

4. How does that relate to what's going on in the organization? 



Source: Bikson et al.. Protocol for Users 

5. What was the department trying to accomplish in converting to online 
computer use? 

Source : Morell 

6. How did you obtain the first PC used in your work group? Please 
circle the most appropriate choice from the list below. 

1. standard purchase requisition 

2. on loan to perform & special project 

3. discovery of equipment not being utilized by others 

4. PC was available when I began my present job 

5. allocated without specific request from my office 

6. other (please explain) 

7. don't know 

Note 1, The items in this question were developed from a previous set 
of open-ended interviews with a subset of the respondents. Thus, the 
response categories reflect the various ways in which PCs were actually 
obtained in this setting. 

Note 2, This question does not reveal the specific purpose for 
acquisition, but it does provide information on whether special efforts 
were made to obtain the equipment. As an example, consider the 
difference in motivation for choices number 4 and 5, versus choices 1 or 
3. The first set implies a passive receipt of the PCs, while the second 
set implies special efforts to obtain microcomputer technology. 
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KEY ACTORS, CRITICAL MASS 

Successful implementation depends on an innovation's having 
supporters in key parts of the organization. On a micro-level, this 
means support from the leaders of work-groups and of the other small 
organizational subunits with immediate influence over the work tasks of 
end-users. 

On a wider organizational scale, it means having active support 
from quarters within the organization who have a vested interest in the 
innovation. Examples of such groups include: planning bodies, groups 
with control over how money will be spent, procurement bureaucracies, 
information system groups, and higher command levels. 

On each level, the greater the number of constituencies sampled, 
the greater the ability to understand implementation dynamics. A 
surface analysis would ask respondents to rate the extent of support 
from their immediate superior and from their organization. A more 
in-depth analysis would probe for the respondents' vested interests in 
seeing the innovation succeed (or fail) , and what was done to further 
that interest. Existing sample questions are all on the work- group 
level. 

Swrpg; Fleischer and Morell m 

1. To what extent does your immediate supervisor encourage your use of 
OA? 

1. totally 

2. very much 

3. some 

4. a little 

5. not at all 

2. To what extent do your p&ers at work encourage your use of OA? (Same 
response categories as above.) 

3. To what extent do your subordinates encourage your use of OA? (Same 
response categories as above.) 
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SUPPORT FOR TRAINING 

SPWrgg; Bikson et al.. Protocol for Users 

1. Now I would like some information about training from your 
viewpoint. 

a- Was learning to use the computer system voluntary for you and 
others in this department? (Probe: Does it differ for 
different lasers?) 

b- What's the main goal of the training program, as you see it? 
(Probe for concrete operations vs. model -based understanding.) 

2. After initial training, about how long did it take before you were 
up to speed on the computer and using it regularly for your work? 

3. Can you describe any formal follow-up support for using the system 
(e.g. useful documentation, reference manuals, online help, whether 
error messages help the user correct his/her mistakes)? 

4. What about informal support? 

5. Long term learning: 

a- What are your opportunities for advanced learning and 
development? (Note whether employee has had any learning 
beyond initial training and whether he/she has pursued or will 
pursue it outside of the firm.) 

b- Have you needed any additional training for new equipment or new 
software acquired by the department? (If so, how wus it 
handled?) 

6. Overall, how satisfactory is the support for learning in this 
department? 

* 

Source: Bikson et al.. Protocol for Managers 

7. Tell me about the training conducted for the computer system. How 
long is initial user training in your department? How much was the 
department involved in designing the training? 

8. After initial training, about how long does it take for an employee 
to ure the computer as a regular work tool, or get up to speed on 
the system? 

9. Is formal follow-up support provided? (Describe) Note instances vf 
useful documentation, reference manuals, on-line help, and whether 
error messages are clear enough to correct the user. 

10. Describe any informal support 
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11. Are there opportunities for advanced learning? (Describe) Note 
whether employees pursue training on their own, outside the firm. 

12. When new equipment or software is e quired how is retraining 
handled? (How often do they expect to retrain?) 

13. About what proportion of the operating budget does the department 
spend on training/staff development? Does the budget provide for 
follow-up or retraining? 

14. Overall, how satisfied are you with learning support? 
S PWrcg; — Blkson et al.. Protoc ol for Technical Evnert- 

15. General description of training, from a Technical Resources 
viewpoint . 

16. After initial training, is formal follow-up support provided? 
(Describe) (Note instances of useful documentation, reference 
manuals, online help, hotline, and whether error messages are 
clear enough to correct the user.) 

17. How does the organization view long-term learning? 

a- Are advanced learning opportunities currently available? (Note 
whether employees pursue training on their own, outside the 
firm.) 

b- When new equipment or software is acquired, how is retraining 
handled? How often do you expect to have to retrain? 

18. In relation to the total cost of converting to interactive systems 
use, what sorts of proportions do the following represent 
(approximately) : 



hardware 
software 
modifications 
implementation 

documentation and training support. 
Source: More 11 



19. Please briefly describe what knowledge you had (if any) about 
computers prior to coming to your present job. 

20. What methods have you and your staff made use of to learn about 
personal computers. Please use the following scale to rate each 
method listed below. 



1- A vgrx important contributor to our knowledge 0 f PCs 

2- An tmpprtfflt contributor to our knowledge of PCs . 

3- Has made ££2B£ contributions to our knowledge about £Cs 

4- A Sing! contributor to our knowledge about ££& 

5- Has had fllfflpgt no impact on our knowledge of PCs . 
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Very in- Impor- Some con- Minor con- Almost no 
PPrtailt taDt trlbution trlbution Impact 

20.1 Formal training 
(courses, work- 
shops, etc.) 

20.2 Interaction with 
informed friends 
or colleagues 

20.3 Self directed 
learning (per- 
sonal reading, 
practice with 
PC, etc. 

21. Now that you told us about the training you have used, please give 
us some information on which sources of information you expect will 
be important over the next few months. Again, use the scale we 
outlined above. 

Same items and response categories as above. 

22. Most people would like more formal training for themselves and 
their staff than they actually use. Using the scale below, 
please let us know the major barriers to using formal training, 
as you see them. 

1 - Training is not available on the specific topics we need. 

2- Training is available, but requires people being away from 
their work for too long a period of time. 

3 - Scheduling is a problem, training is not available when we 
need it. 

1- The mSSi important impediment to training. 

2- A rniQH impediment to training 

3- A problem, but hq£ a serious one 

4- A very min2i problem 

5- £g£ a Problem at all. 

6- Don't know, or no opinion. 
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Most Major Problem, Very Not a No 

impor- impedi- not ser- minor prob- opin- 
tftllt (1) ment m l W g (3) problem (4) lem (5) Ion (6) 



22.1 No training 
on specific 
topics ve 
need 

22.2 Training 
unavailable when 
needed (schedul- 
ing problem) 

22 . 3 Training 
requires too 
much absence 
from work 

22.4 Not 

enough advance 
notice of 
courses 

Source: Rotman 

23, How were you trained for WANG word processing? (check as many as 
apply) 

specify course (s) or topic (s) 

Courses at WANG laboratories 
Programmed instruction books 
WANG instructor at your company 
Company 9 s in-house trainer 
Self- instruction, experimentation 
From other users or key operators 
From tapes or audio -visual methods 
At a business school 
Other (please specify) 

24 • For the system you use most often, did you receive formal training 
(with an instructor) before the system was installed? 
YES NO 

If yes, please specify 
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25. For the system you use most often, did you receive formrl training 
(with an instructor) after the system was installed? 

YES NO 

If yes, please specify 

26. Was the system already in use when you received your training? If 
YES, how long had it been operational? 

years, months. 

27. I am interested in finding out what formal training (with an 
instructor) you have received on the system you are currently 
using. Please indicate which of the following courses you have had 
and what kind of WANG system you operate. 

Course System OIS Vg_ AL LIANCE OTHER ( ) 

Intro to word processing 

Advanced functions 

Advanced word processing 

Decision processing 

Other 



28. To the best of your recollection, please rate the quality of 
Training on any of the courses you indicated you had taken. 

P°Pr fail G?p4 Excellent 

Intro to word processing 

Advanced functions 

Advanced word processing 

Decision processing 

Other 



Souyce; Fleischer and Morell m 

If you have received any formal training in the use of your OA please 
answer the following questions. "Formal" training is any training that 
involved a trainer who does training as part of his/her job. If you 
have not received and formal training, please skip this section. 

29. How many training courses have you attended related to your present 
OA equipment? . 

Please answer the following questions with regard to the most 
recent course you attended. 

«5 
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30- Who paid for the training? 

1. the company/agency you work for 

2. you did personally 

3. vendor 

4. don't know 

5. other (describe) 

31. Where was the instructor from? 

1. supplied by vendor 

«i. outside consultant 

3. staff from your company/agency 

4. don't know 



32. How knowledgeable was the instructor? 

1. extremely knowledgeable 

2. very knowledgeable 

3. somewhat knowledgeable 

4. a little knowledgeable 

5. not at all knowledgeable 

33. How long did the training last? hours of training 

34. Please rate the amount of "hands on" (direct experience using the 
equipment or application) time you had during the training (as 
opposed to lecture and discussion time). 

1. much too much hands on time 

2. a bit too much hands on time 

3- just the right amount of hands on time 

4. a bit too little hands on time 

5. much too little hands-on time 

35. How satisfied were you with the training you received? 

1. completely satisfied 

2. vory satisfied 

3. somewhat 

4. a little satisfied 

5. net at all satisfied 

36. Have you used the information you gained in rhis course to train 
others in your unit? 

1. No 

2. Yes (how many others?) 



37. Please briefly describe airy problems you experienced with the 
training you received. 
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EXPERTISE (OF OTHERS) AVAILABLE TO END-USERS 
S&UIfiftj Bikson at al.. Protocol for Managua 

1. In nost workplaces, a technology "expert" usually emerges. This is 
the person whoa most people go to when they have a question or 
problem. Do you have an "expert"? What would you do (did you do) if 
he or she left? . 

2. Do you have access to a "local expert" on personal computers? 

Definition of a "local expert*: A person who works directly for you 
who applies specialized knowledge of personal computers to needs that 
are defined by you. 

if YES go to q# 9 
if NO go to q# 

3. Please use the following chart to tell us what that "local expert" 
does. (If you have more than one person who fills this role, please 
fill in the chart with l's, 2's and so on to refer to these different 
individuals • ) 



4.1 Writing programs/ 
developing special 
applications 

4.2 Training and giving 
advice 

4.3 Analyzing information 

4.4 Other (please specify) 

5. Does your "local expert" do other tasks besides the computer related 
work referred to above? 

if YES go to q# 6 
if NO go to q# 

6. Please briefly describe the other work this person does. 



Major use 
of expert's 
time 



Secondary use 
of expert's 
time 



All lesser 
amounts of 

time 
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7. Please check the category below that beat describes how you obtained 
the services of your local expert. 



Student intern (1) 

Regular Navy personnel allocation system (2) 

* Civilian Navy employee , (3) 

Loan from another group (4), 

Paid consultant (5) 

Other (please explain) (6) 



8. Are there any people who fulfill the "local expert" role for you- 
but who do not work directly for you? (As an example, someone in a 
computer support center or in some other group may be helping to 
develop a custom program for your use.) 

if YES go to q# 18 
if NO go to q# 

9. Same wording and response categories as Q #10 

10. Please briefly describe how you obtained the services of this 
expert. 

11. I am interested in the ways you resolve problems with your system 
when they occur. Please indicate how often you use each of the 
following when you need help. 

£&v£K Rarely Sometimes Often 

Read manuals 

Call VANG hotline 

Ask another usor for help 

Call VANG support arulyst 

Ask key operator/supervisor 

Experiment • try things out 

Other 



12. How well do you feel each of the following problem solving 
techniques works? 

1 • Always provides an acceptable solution. 

2 - Usually provides an acceptable solution. 

3 - Occasionally provides an acceptable solution. 

4 - Rarely provides an acceptable solution. 

5 - Never provides an acceptable solution. 

6 - This source of assistance is not available to me. 
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Always Usually Occasion- Rarely Never Not avail- 
accept- accept- ally accept- accept- accept- able to 
flfrl? able aole able able to me 

Read 

manuals 

Call WANG 

hotline 



Ask another user 
for help 



Call WANG support 
analyst 



Ask key operator/ 
supervisor 



Experiment - try 
things out 



Other 



13. If you normally get hel : > from another user, is he/she in your 
department? YES NO 

14. Approximately how many WANG users are there in your company? 

a. fewer than 10 

b. 10 - 25 

c 26 - 50 

d. 50 - 100 

e. over 100 

15. How many WANG users are there in your department? (Answer only if 
applicable. ) 

a. fewer than 10 

b. 10 - 25 

c 26 - 50 

d. 50 - 100 

e. over 100 

Note: Questions 5 and 6 are useful in estimating both the possible 
"richness 11 of informal support networks, and the amount of expertise an 
organization may need to support its employees. 

16. Is there a person assigned to formally coordinate the WANG 
equipment in your company? YES NO 

Source: Fleischer and Morell m 

17. How many others in your work group use the same OA (same or very 
similar equipment) that you use? 

18. How many others in your work group use OA for the same applications 
as you? 
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19, An Information Center is a special unit designed to assist users; of 
OA and other information technologies in the use of that equipment. 
To your knowledge, is there an Information Center or something 
similar in your company/agency? Y ES NO 

If NO, skip to question 20. 

If yes, please answer the following. 

A. About how many times have you used this centar's services in the 
past year? 

B. What Center resources have you used? (Check all that apply.) 

19.01 equipment or software manuals 

19.02 equipment to borrow 

19.03 software to borrow 

19.04 consultation help with purchasing decisions 

19.05 consultation fcelp with use of OA 

19.06 programming for special projects . 

19.07 programming for everyday projects 

19.08 training 

19.09 advice on OA purchases 

19.10 other (describe) 

20. There are many potential sources of help and support for OA use. 
Please circle the number which indicates how important the 
following sources of help and support are to you. 

Extremely Very im- Somewhat A little Not im- Not 
important portant portant important portant available 
(1) (2) (3) (4) (5) (6) 



ERLC 



20.1 Formal expert 
in your work group 
(helping is part of 
oersons job 1 


2 


3 


4 


5 


6 


20.2 Informal expert 
in your work group 
(helping not part of 
oersor/s lob 1 


2 


3 


4 


5 


6 


20.3 Informal expert 
outside of your 
v.ork group (but in 
vour companv) 1 


2 


3 


4 


5 


6 


20.4 Formal users 












proup l 


2 


3 


4 


5 


6 


20.5 Informal users 












prouD 1 


2 


3 


4 


5 


6 


20.6 Information 












center 1 


2 


3 


4 


5 


6 


20.7 Outside con- 












sultant (paid to 
helt>> 1 


2 


3 


4 


5 


6 


20.8 Friend outside 












your company/agency 
(not naid) 1 


2 


3 


4 


5 


6 
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END USERS' EXPERTISE 
SQVrcg; Morell 

1. Please briefly describe what knowledge you had (if any) about 
computers prior to coming to your present job? 

Source; Rotman 

2. Before accepting this job, did you use any of the following? 

rarely - a few times a year 
sometimes - a few times a month 
often - every week 

Never barely Sometimes Often 

2.1- Electric typewriter 

2.2- Electronic typewriter - 

2.3- Mag card machine 

2.4- Microcomputer 

2.5- CRT 

2.6- Video game machine 

2.7- Other word processor 

3. Do you own any of the following? 

YES NO 

3.1- Electric typewriter 

3.2- Electronic typewriter 

3.3- Mag card machine 

3.4- Microcomputer 

3.5- CRT 

3.6- Video game machine 

3.7- Other word processor 

Source: Fleischer and Morell (1) 

4. Prior to graduation, did you ever take a high school or college 
computer course? 

1. NO 

2. YES 

5. Do you have a computer at home? 

1. NO 

2. YES 

5.1- If YES - do you personally use it? 

1. NO 

2. YES 
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ADAPTIVE PLANNING/USER PARTICIPATION 

gwr<re; — Blkson et ml.. Protoc ol for Managers 

1. When did this department first change to online? Tell me something 
about the process. When did planning for this change begin, and 
what was the general timing and pacing of the effort? How were the 
events sequenced? What was the general approach? The focus? 

2. Tell me how you went about implementing the technology. How 
detailed was the planning? Was the process flexible or 
experimental? Was there more concern with technical issues or with 
human issues? 

3. We're interested in who had decision making responsibilities in a 
number of domains. Consider the following group of people - 
executive management, technical resource persons, human resource 
persons, department managers, and users - who made decisions about: 



hardware 
software 

implementation process 
work environment issues 
employee impacts 



(Note whether these actors were coordinated in the decision process; and 
if so, how? Probe for balancing of varied inputs, and especially for ' 
participatory processes.) 

Source: Fleischer and Morell m 

4. How much influence do you have concerning the purchase of OA 
equipment in your work group? 

1. almost complete control 

2. quite a bit of influence 

3. some influence 

4. a little influence 

5. almost no influence 

5. Kow much influence do you have concerning the purchase of software 
or applications for OA in your work group? 

Same scale as above. 

6. How much influence do you have concerning the use of OA in your 
work group? 

Same scale as above. 




71 



INCENTIVES FOR USE 

One aspect of incentives for vise - those which exist at the 
beginning of an implementation effort - can be discerned by using the 
same information collected under "reason for adoption. " As 
implementation proceeds, the focus shifts to whether the advantages of 
using the new technology outweigh the problems that inevitably accompany 
any such implementation effort. Most studies address this issue 
indirectly, by collecting information on the positive and negative 
impacts of implementation. 

A second aspect of "incentives for use" is the special case where 
micro -computers are used to make up for deficiencies in the respondents' 
overall information system environment. Because information on this 
type of use cannot be obtained without specifically asking for it, 
examples of appropriate questions are given below. 

Source: Morell 

1. We are interested in what information systems you use other than 
ones which reside on your personal computer. With regard to these 
systems, please fill out the chart below. 

Name of Major uses Major disadvantages 

gystgnt of system of system (if any ) 

system #1 



system #2 



system n 

2. If you use personal computers to augment information from any of 
the above systems, or to make up for any of their deficiencies, 
please tell us how. 

system #1 
system #2 



system n 



c 
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DESCRIPTIONS OF WORK GROUPS AND THEIR ORGANIZATION 
Source: Morell 

1. Please describe your job* 

2. How many people work for you, and what do they do? 
Source: Fleischer and Morell m 

3. Please check the description that best characterizes your 
company/agency . 

1. government 

2. education 

3. health related products or services 

4. financial services 

5. consumer products or services 

6. products or services to business or government 

7. other 

Note: Lists such as appaar above can be modified to obtain data on 
sub-parts of a larger organization. 

4. How many employees (of all types) does your work group have? 

5. How many managers are part of your work group? 
SofflTW Fleischer and Morell m 

6. Which of the following best characterizes your present job? 

1. first line supervisor 

2. middle management 

3. senior staff 

4. senior management 

5. executive 

6. other (describe) 

Source: Fleischer an d Morell m 

7. Job titles: (Use approximate or descriptive titles if you are not 
sure of precise wording.) 

7.1 your own title 

7.2 your direct supervisor's title 

7.3 titles of people, you immediately supervise 

1- 
2- 
3- 
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8. How many people do you directly supervise? 

9. What are your major responsibilities? 

10. What are your company's major products or services? 

11. Approximately how many employees does it have? 
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ACTIVITY WITHIN WORK GROUPS - USE OF OFFICE AUTOMATION 
Source; Bikson et al.. Protocol for Technical Expert 

1. What percentage of your employees are computer users? 

2. What percentage of employees have workstations? 

3. What kinds of workstations do you have here? 

4. Do terminals have local processing power (if applicable)? What 
kinds of processing units (mainframes, minis, micros, or multiples 
of these) drive the terminals (if applicable)? 

5. How many different hardware vendors are represented in these 
departments? Where does the software come from? Is the software 
customized? 

6. Whet are the major applications in use? What tasks are they used 
for? 

7. In general, when employees interact with the computer system, does 
the process unfold automatically, or do the users guide the process? 

8. What capabilities are there for user modification (e.g., 
user-modifiable menus, user-definable keys, an end-user programming 
language?) 

9. Is computer-based mail in use? Who communicates with whom and for 
what purposes? 

Swrg?; Bikson et al.. Proto col for Managers 

10. What kind of workstations do you have here? 

11. What major applications (tasks) are they used for? 

12. Do you have computer-based mail? Who is on it so far? What 
percent of the eirployees use it? What do they use it for? 

Spurce; Bikson et al.. Protocol for Users 

13. When did you first start using a computer here? 

14. Do you have your own workstation? (If not, how is it shared?) 

15. Do you have any procedures for modifying the system (e.g., 
user -modifiable menus, user-modifiable keys, an end-user 
programming language?) If so, do you get a lot of use out of 
them? (Explain) 

16. Do you use computer-based mail? Uith whom do you mainly 
communicate? (Probe for informal as well as task-related 
communication.) 
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17. How much time, in a typical day, do you spend working at a 
workstation? (hours or % of total time) Is it continuous? 

Spum; Morell 

18. We arc interested in the amount of time that you and your staff 
invest in using PCs. Please fill out the chart below with a 
general estimate of che number of hours that you and your staff 
have put into PC related work over the past month . 



Approximate Approximate 
hours - mvself hours - staff 



Developing programs of 






soecial annlications 


ris.on 


(18,021 


Entering data into 






the PC 


(18. 03) 


as ,04i 


Word processing/ 






document production 


(18.05) 


as. 06) 


Using data based 






management orograms 


(18.07) 


a*. 08) 


Using spreadsheet 






Droerams 


as. 09) 


(18. 1C) 


Training (both formal 






and self directed) 


as. id 


as. 12) 


Other 






falease specify) 


as. 13) 


(18.14) 



19. Do any of your staff engage in any of the above listed activities on 
personal time? If so, please briefly explain how much effort is put 
in, by whom, and to what purpose. 

20. What kind of word processing systems do you primarily use in your 
department? (Check as many as apply.) 

a._0IS 40/50 b. 0IS 105/115 c. QIS 130/140 

d.__WP system e. Alliance f . WANG personal cc -uters 

g.___Wangwriter h. VS series i. other( 1) 

?J During the past month, approximately what percentage of your time 
did you spend using your WANG word processing system? 

a. less than 10% b. 10 - 25% c. 26 - 50% 

d. 51 - 75% e. 76 - 100% 
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22. Do you have any of the following options on your WANC system? If 
so, during the past month, how often did you generally use each 
one? 

rarely « a few times a year 
sometimes - a few times a month 
often - every week 

Don't Never Rarely Sometimes Often 

hflY? MS? use use use 

22.1- List processing 

22.2- BASIC 

22.3- Readability Index 

22.4- Dictionary 

22.5- Telecommunications 

22.6- Mailway ^^^^ ZZZ 

22.7- WISE 

22.8- Glossary 

22.9- Math 



23. Following are tasks that can be performed on a word processing 
system. Please indicate which tasks you perform and how often. 

MgvfeE Rarely So metimes Often 

23.1- Create letters, memos, etc. 

23.2- Prepare financial reports [ m 

23.3- Print envelopes ' 

23.4- Prepare/print labels \ 

23.5- Create "merge" documents 

23.6- Use pre-printed forms 

23.7- Create/maintain lists 

23.8- Produce contracts, etc. 

23.9- Create newsletters, etc. 



Source: Fleischer a nd Morell m 

24. Circle your best estimate of how often you used the following OA 
equipment in the oa st month . For example, if you used a piece of 
equipment about every day in the past month, circle a "1" by that 
equipment . 
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More than About Once or Available Not 
once per once per a few but not avail- 

Pally aantb manfli times used able 

24.01 Electronic 

mail 1 2 3 4 5 6 

24.02 Video confer- 
encing equip- 
ment 1 2 3 4 5 6 

24.03 Micro or per- 
sonal computer 12 14 5 6 

24.04 Terminal access 
to mainframe or 

minicomputer 1 2 3 4 5 6 

24.05 Printer 1 2 3 4 5 6 

24.06 Plotter 12 3 4 5 6 

24.07 Modem (any 
phone access 

to a computer 12 3 45 s 

24.08 Stand alone word 

processor 12 3 4 5 6 

24.09 Local area 

network 1 2 3 4 5 6 

24.10 Other 

(describe) 1 2 3 4 5 6 



25. What types of OA do you have available to you in your personal 
workspace (i.e. at or near your desk?) (Check all that apply.) 

25.01 Electronic mail system 

25.02 Video conferencing equipment 

25.03 Micro- or personal computer (PC) 

25.04 Terminal access to mainframe or minicomputer 

25.05 Printer 

25.06 Plotter 

25.07 -Modem (for telephone access to a computer) 

25.08 _____ Stand alone word processor 

25.09 Local area network 

25.10 Other (describe) 
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26. Circle your best estimate of how often you (or your assistant) 
used the following OA applications in the past month . 

More than About once Once or Available Not 
once per per a few but not avail - 

fiftilX VCtK Mgak times used 

26.01 Word 



processing 


1 


2 


3 


4 


5 


6 


26.02 Accessing 














data bases 


1 


2 


3 


4 


3 


6 


26.03 Managing/ 














manipulating 














data 


1 


2 


3 


4 


5 


6 


26.04 Spread- 














sheet 


1 


2 


3 


4 


5 


6 


26.05 Modeling/ 














forecasting 














(other than 


* 












spreadsheets) 


i 


2 




4 


5 


6 


26. C6 Statistical - 














analysis 


i 


2 


3 


4 


5 


5 


26.07 Graphics 


i 


2 


3 


4 


5 


6 


26.08 Electronic 














mail 


i 


2 


3 


4 


5 


6 


26.09 Tele- 














conferencing 


i 


2 


3 


4 


5 


6 


26.10 Account- 














ing 


i 


2 


3 


4 


5 


6 


26.11 Training in 














work related 














activities 


i 


2 


3 


4 


5 


6 


26.12 Other 














(describe) 


i 


2 


3 


4 


5 


6 



SpVHttg; Fleischer and Morell m 

27 . Do you use office automation in a manner that involves your 
actually using the equipment yourself? YES NO 

(If NO, skip to question # ) 

28. Please describe the last two times you remember personally using 
office automation equipment. 



use #1 



use #2 
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29. Does your answer to question #28 represent the most frequent 
uses you personally made of office automation? 

YES* NO 

(If YES, skip to question # ) 

30. What is the most frequent use you personally make of office 
automation? 



31. Besides the use listed in question #4, are there other common 
uses you have for office automation? 

YES NO 

(If N0 t skip to question # ) 

32. Please briefly describe some of these other common uses for 
office automation. 



use #2 



Note: The above sequence (27 - 32) is repeated with slight wording 
changes to address use by an immediate superior and an immediate 
subordinate of the respondent. 
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ACTIVITY WITHIN WORK GROUPS - SATISFACTION WITH OFFICE AUTOMATION 
Source: Bikson at ml,. Protocol for Technic al Extort 

1. Is the equipment reliable (crashes, downtime, slowness)? Can you 
estimate how much lost time costs the organization as a result of 
system malfunction? 

2« Overall assessment of system (what's working well, what's still 
missing.) 

Source: Bikson et al.. Protocol for M*n«g»r« 

3. How reliable has the system been (crashes, downtime)? Can you 
estimate how much time your department loses as a result of system 
malfunction? 

Source: Bikson et al.. Protocol for Usera 

4. We're interested in any problems you may have vith computer 
reliability or availability , for example: 

Can you easily get to use the aquipment or software you need? 

Is the response time slow/adequate " c ast? 

Does the equipment crash or have a long downtime? 

5. Overall, from your perspective, what works well in the system? What 
doesn't work well? 

SamsAi Batman 

For each of the four questions below, check which applies. 

6. Noise level while you are on WANG 

a. so quite, I hear no disturbing sounds 

quite enough to get my work done without being: disturbed, 

c. noisy enough to make working very difficult, but not 

impossible 

d. so noisy, I can't do my work 

7. Interruptions while you are on the WANG 

a* few, if any 

b. an acceptable amount; they occur, but don't keep me from 

doing my work 

c. enough to make working difficult, but not impossible 

d. so many, I can't get my work done 
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8. Temperature of the room or area in which you use WANG 



a. too cold 

b. cool, but bearable 

c. not too hot or too cold, just right 

d. warm, 

e. hot enough to be uncomfortable 

9 . Privacy you have while on WANG 

WANG nool (a number of users doing nothing but WP in one 

area 

b. total privacy (a separate room) 

c. come privacy (partition in a larger area, separate part 

of a room) 

d. no privacy (users workJng in the "ritual 11 area) 



Source: Fleischer and Morell QJ 

10. Many times a piece of OA equipment is available fcr the use of 
several different people in an office. Because of competing 
uses it is sometimes difficult to get access to that equipment. 
How series is this problem of "access for you? 



1. ;an extremely serious problem 

2. a very serious problem 

3* a slightly serious problem* 

4. only a mild problem 

5. not at all a problem 



Source: Morell and Leemon 

11. Following is a list of some of the things that are often done by 
hand that, theoretically, can be performed on the Star. Please 
check those that you frequently, not always, do by hand rather 
than on the Star. 



a. sketches 

b . ; drawings 

c. presentation material 

d. text revisions 

fc* cut and paste specifications and documents 

f . marking -up documents 

g. mail functions 

h. ^. document logging 

i. forms control 

j . other 



12. Please explain in one or two sentences your major reasons for 
wanting to do some of the above tasks by hand. 
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13. Do you believe the Star is being used to its full potential? 

YES NO 

If you checked "no", please answer the following question. If "yes" 
skip to question 15 . 

14. Please describe very briefly the ways in which utilization of the 
Star could be made better. 



15. Please describe the two or three things you like most about the 
Star. 



16. Please list two or three things that you dislike most about the Star 
that you would like to see changed, (for example, in terms of 
capacity, availability of software, etc.) 



Source: Fleischer An d Morell m 

17. Are there any applications you would like to use office automation 
for, but cannot in your present job? Y ES NO 

(If NO, skip to question # ) 

18. What are the major obstacles to the use you mentioned in the 
previous question? (ex: access to hardware, available software, 
training, approval of company, etc.) 
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QUALITY AND QUANTITY OF PRODUCTION 

For two reasons, most studies of the impact of OA do not explicitly 
separate questions about the quality and quantity of production. First, 
the decision as to what is a quantitative or qualitative difference is 
often made by the researcher, and is dependent upon the circumstances 
the researcher is trying to explain. As an example, consider findings 
about using microcomputer based graphics. If a respondent's job is to 
produce graphs, an increased number of graphs may be seen as an issue of 
production or productivity. But if the respondent's job is to analyze 
data and present briefings, an increased use of graphs might be seen as 
a qualitative improvement. The decision is the res archers' and is 
based on a knowledge of the work context, 

Second, because interpretations about quality and quantity are often 
based on data about what respondents actually do, the primary objective 
is to ask questions in a way that will allow the respondent to supply 
necessary detail. Often the best way to get that detail is to ask for 
specifics about work activities and the demands made upon people. 
Questions of that nature do not lend themselves to overt distinctions 
between qualfty and quantity. Many of the examples presented below will 
illustrate this point. 

Just as issues of quality and quantity are often mixed in the sane 
question (or set of questions) , so too is there a mix of questions about 
the impact of OA and judgements about the value of that impact. As with 
the qualitative and quantitative distinction, the judgement about value 
is often left to the interpretation of the researcher. As an example, 
an increase in the amount of information reported at briefings may be 
positive or negative, depending on whether it leads to better decisions 
or more confusion. Also, the requirements of good question order 
sometimes require not separating questions about th*. value of a change 
from question" about the nature of the change itself . This too will ba 
illustrated ' . the following examples. 

Source: Bikson et al. . Protocol for Technical Experts 

1. What has been the impact of information technology on: 

a. productivity (how assessed?) 

b. labor costs (how assessed?) 

c. on other efficiency goals (how assessed?) 

1. Do any of the focal departments figure predominantly in these 
effects? 

2. What have been the impacts oh other strategic organizational goals? 
Source: Bikson et al.. Protocol for Managers 

3. What changes have you seen over time in efficiency and effectiveness? 
(Effipibhcy usually means fewer people; effectiveness means doing a 
better job.) (Probe for changes in quality, speed, quantity and so 
on.) 
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4, Have these performance changes made any difference in the performance 
of the organization? (e.g., productivity, other strategic goals?) 

Source: Morell 

5, Describe how you use your PC (use #1). 



6. Which of the following statements best characterizes the role of the 
personal computer in this task? 



1- This task would be done about as well, even without a personal 
computer. 

2- The personal computer is some help in dcing this task. 

3- The personal computer is a great help in doing this task. 

4- It would not be practical to do this task without the help of a 
personal computer. 

7. Using the scale below, please rate the consequence of this task for 
the Navy. 



1- This is extremely beneficial for the Navy. 

2- This has some bene fit for the Navy. 

3- It makes nfi difference to the Navy whether a personal computer 
is or is not used for this task. 

4- This is somewhat detrimental for the Navy 

5- This is extremely detrimental for the Navy. 

6- I don't have enough information to form an opinion. 



Extremely Some No Somewhat Extremely No 
bene- bene- dif- detri- detri- opin- 
ficial fit ference mental mental ion 

Hi — -121 Q) m (6) 

Dollar 

savings 

New information 

for planning 

Speed o m 2 obtain- 

ing information 

Number of people 

needed to do 1obs 

Coordination among 

people or yrnups 

Response to re- 

quests from above 

Information >Ior 
people of equal or 

lower rank ; 

The above set of questions can be repeated as often as, necessary if the 
respondent has mojre than one use for OA. 
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SWrg?; Fleischer and Morell m 

8, Given all the times you can ever remember personally using office 
automation in your pr2sent job, what use had the greatest impact for 
you or your company? 

9, Why was the use described in question #8 so important? 

Note: These questions were asked three times, referring in turn to 
the respondent, his or her boss, and a person he or she supervised. 

gpurgg; Morell and L esmon 

10, Of all the tasks that you used the Star for, what tasks do you think 
are the most important? Please explain why. 



11. Sine 4 you have had access to the Star have zhere been any changes 
you've observed in the frequency of written documents? 

YES NO 

12. If yes, please describe what kinds of changes. 



13. Since you have had access to the Star have there been any changes 
you've observed in the length of written documents? 



YES NO 

U. If yes, please describe what kinds of changes. 



15. Do you find that the Star helps you to accomplish your old tasks 
faster? 

YES NO 

If yes, please answer the following. If no, skip to question 

16. Now that you have more time, how do you use it? (Check as many of 
the following as apply.) 

a. I do more old tasks. 

b. I do more of the same task3 better. 

c. I perform new kinds of tasks. 

d. I do some of thfc same tasks differently. 

e. I have more time to think about old tasks. 
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17 • It is possible that having access to the Star may have an impact on 
several areas in your work life. How would you rate the Star's 
impact on the following: 

Very Very 
No little Minimal Moderate Great great Don \ 
jfflP?g£ iffPflCt impact impact impact impact know 

17.1 - Preparing 

reports s 

17.2- Monitoring 

schedules 

17. 3 - Tracking 

projects 

17. 4 - Meeting 
customer re- 
quests . 



17. 5 -Meeting 
deadlines 



17.6-Meeting 
in-house re- 
quests 



17. 7 - Acquiring 
new 'tiugineas 

17.8- Finding 
effective 
solutions to 
engineering 

scbMshs 
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ADAPTABILITY AND CHANGES IN WORK GROUPS 

As with the concepts of "quality and quantity of production, " it is 
difficult to separate questions about a work group's functioning from 
questions about its adaptability to new circumstances. As an example, a 
new microcomputer may increase a group's use of dBase files in lieu of 
manual record keeping. That is a change in functioning which may also 
affect adaptability, as mediated through the new-found flexibility of 
data access. In such a case, the researcher must integrate information 
on use of dBase with other knowledge about how the work group changed, 
and make a judgement about adaptability. Even though some information 
about adaptability may come from direct questions about it, other 
relevant information must be inferred from descriptive data on work 
group functioning. Thus from a practical point of visw, questions about 
"work group functioning" and "work group adaptability" are integrated. 

SWP?g; Biksori et al.. Protocol for Te chnical Expert 

1. How has the work changed as a result of the technology 
implementation? (e.g. managers, professionals do more keyboarding; 
secretaries have more tasks; new tasks are designed; new processes 
for doing work adopted.) 

2. Have people invented new ways of doing their work as a result of the 
technology? 

Source: Bikson et al. . Protocol for Technical Expert 

3. How has the work changed as a result of the technology 
implementation? (Probe for changes in managers, professionals, 
secretaries, other clerical employees.) 

a- Any changes in the ways you organize the work (work- flow/ 

in f orma t ion - r low? ) 
b- Any changes in the hours of work or place of work? (Probe for 

home work and work contracted out - especially offshore.) 

4. Have people invented new ways of doing their work as a result of the 
technology? 

5. As of now, could employees go back to the old way of doing things? 
Spyrgg; Btkson et al.. P^^ col for Technical Expert 

6. Now I would like to ask you about any changes in your work - the 
activities, what you do - related to the system. (Probe for 
expansion vs. constriction, and for redistribution of work.) 

a« Changes in vi iety/variability? 

b- Changes in control over your work, especially in what is left to 

your own judgment? 
c- Do you have new tasks and responsibilitias, or do you do the 

same job with a new tool? 
d- Changes in type of supervisory support (Probe for machine pacing 

or monitoring of work.) 
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7. As of now, could yon -or other people in your department go back to 
the old way of doing things? 

8. Has using the computer affected your job in official terms, e.g., 
Your job title? New job description? Computer* related pay increase? 

Other? 

Source: Fleischer and Morell (1) 

9. Office automation can fit into a manager's worklife in many different 
ways. One important way is in how managers solve problems. Below 
are three different types of problem solving situations that you 
might encounter in your work. Please briefly describe ;a recent 
example in which you dealt with each type of problem, solving 
situation. 

CUT AND DRIED Cut and dried problems are those in which thers is a 
clear set of formal or informal rules which make a decision 
relatively straight forward, once the proper information is known. 
For example, many scheduling, inventory, and credit decisions can be 
made in this way. 

Example : 



ANALYT IC Analytic problems are those which are sufficiently complex 
to prevent a set cf rules from acting as a routing formula to solve the 
problem. For example, a decision about whether to market a product in a 
particular location involves formal analysis, but is not determined by a 
formula • 

* 

Example : 

UNSTRUCTURED Unstructured problems are those for which there are no 
formal guidelines and few tested decision making criteria. There nay be 
a lot of information available, but no reliable guidelines for using the 
information. Hiring top managers and long range planning are some 
problems that usually fit into this category. 

Example : 



For each example that you gave, please indicate > using the following 
numbers . how much each of the advantages of OA below helped you to solve 
each problem 

1- helped a very "great deal 

2- helped quite a bit 
3* helped some 

4* helped a little 

5* did not help at all 

6- is not: available to me. 
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a. Easily generate and 
revise text 

b. Quickly and cheaply 
gather information 

c. Work with data 

i bases more easily 

d. Quickly and cheaply 
analyze data 

e . Make routine calculations 
more easily 

f ♦ Test different models 
more easily 

g. Graphically display 
^ analyses 

h. Send messages to others 
without their having to be 
present to receive them 

i. Hold conferences without 
needing to travel 



Cut and dried Analytic Unstructured 

problem problem problem 

9.01 9.02 9.03 

9.04 9.05 9.06 

9 ,07 9 . 08 9 . 09 

9.10 9.11 9.12 

9.13 9.14 9.15 

9.16 9.17 9.18 

9.19 9.20 9.21 

9.22 9.23 9.24 

9.25 9.26 9.27 



10. Has your use of OA enabled you to deal with new problems, 
those which you had never been able to address before? If 
so, please describe what the new problem was and how OA 
helped. 

11. How many new (to you) applications or programs have you 
tried during the past 3 months? 

12. Please think about a recent project that involved significant, 
ongoing collaboration with people outside of your vork group. 
Please describe what role, if any, OA played in this collaboration. 



13. Some managers are frequently involved in collaborative projects such 
as that referred to in question 17, others net so frequently. Over 
the past 3 months, about how many different collaborative projects 
have you been involved in? 

(A collaborative project is a project that has its own budget or 
projected budget.) 



1. none 

2. 1 - 2 

3. 3-4 

4. _ 5 - 6 

,5. 7 or more 

14. Have you ever used any form of OA for work related tasks at 



n 
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home or while traveling? 



1. 
2.. 



NO 

YES - IF YES, What applications >iave you used? 



15. Sines you have been working in your recent job, have you 

observed any impact of OA in the amount of time you spend on 
the following activities? Please circle the number 
corresponding to the anount of change for each activity. 



Much 

more More Same 
time time time 



15.10 Talking on the telephone 

15.20 Preparing reports 

15 . 30 Preparing presentations 

15.40 Meeting with subordinates 

15.50 Meeting with peers 

15.60 Meeting wit? superiors 

15 . 70 Meeting with people from 
outside your organization 

15.80 Using data in decision 
making 

15.90 Analyzing data 

15.10 Thinking 

15.11 Traveling on business 



2 
2 
2 
2 
2 
2 

2 
2 



2 
2 



3 
3 
3 
3 
3 
3 

3 
3 

3 
3 
3 



Less 
time 

4 

4 

4 

4 

4 

4 

4 
4 

4 
4 
4 



Much 
less 
time 

5 

5 

5 

5 

5 

5 

5 
5 

5 
5 
5 



Can't 
sai 

6 
6 
6 
6 
6 



6 
6 
6 



16. Has the time you devote to other activities changed as a result of 
your use of OA? If so, please describe briefly. 

17. Please indicate about how much impact each of these applications has 
had on your work. Circle the number by each application that 
corresponds to one of the following statements: 



1- marked and profound change for the better 

2- major beneficial impact 

3- some beneficial impact 

4- slight beneficial impact 

5- no impact at all 

6- negative consequences have outweighed belief its 

Profound Major Some Slight No Negative 
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Profound Major Some Slight No Negative 
change benefit benefit benefit impact effect 



18.01 Word processing 1 2 3 4 5 6 

18.02 Accessing data bases 1 2 3 4 5 6 

18.03 Managing/ 

manipulating data 1 2 3 4 5 6 

18.04 Spreadsheet 1 2 3 4 5 6 

18.05 Modeling/forecast- 1 2 3 4 5 6 
ing (other than 

spreadsheets) 

18.06 Statistical analysis 1 2 3 4 5 6 

18.07 Graphics 1 2 3 4 5 6 

18.08 Electronic mail 1 2 3 4 5 6 

18.09 Tele -conferencing- 1 2 3 4 5 6 

18.10 Accounting - 1 2 3 4 5 6 

18.11 Training in work 1 2 3 4 5 6 
related activities 

18.12 Other (describe) 1 2 3 4 5 6 
Source: Morell and T^mnn 

19. Since the Star was introduced have you received any additional work 
assignments, or n*w areas of responsibility? 



YES. 



N(> 



20. If yes, please provide an example. 



21; Are there any tasks you are able to accomplish now that you could 
not accomplish without the Star? 



YES. 



NO 



22. If yes, could you give an example? 
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23. If you had to do the work you do now vitnout the Star, how difficult 
do you think it would be? (Check only one of the following,) 



a It would make no difference at all, 

b It would make a minor difference. 

c It would make a moderate difference, 

d It would make a significant difference. 

e It would be an impediment to my productivity, 

f It would be a major problem. 

g I would not be able to perform most of the work that I do. 
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COMMUNICATION 

Swrgg; Mkson et al.. Protocol for Technical Expert 

1. Have there been changes in communication and relationships between 
individuals or between departments? (Describe.) 

2. Do computer-based communications replace memo's, phone calls, 
in-person discussion? (Explain) 

3.. Have, computer-based communications created electronic "islands'* as 
yll as connections? Have they affected the work flow among 
departments or otherwise altered the coordination or structure of 
activities? 

Source : Bikson et al . Protocol for Managers 

4. Does computer-based mail affect communications, such as memos, phone 
calls, in-person discussion, rangp of contacts? Explain? 



Source: Bikson et al.. Protbcol for Users 

5. Have there been changes in communications, interactions, 
relationships among people here because of the computer system? 
(describe.) 

Source: Morell 

6. Please tell us about written reports you have prepared which 
incorporated PC-based information. (We are only interested in the 
more important examples. A complete list is not necessary. 



Report #1 

6.1- Descriptive title of report 



6.2- Frequency of report (as needed, weekly, monthly, etc.) 



6.3- Primary audience 



6.4- Description of contents 



6.5- Did the report contain any information from data sources other 
than your PC? If so, please explain briefly. 

Report #2 > . . The above set of questions can be repeated as 
necessary. 
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7, Please tell us about briefings you have prepared which incorporated 
PC-based information, (We are only interested in the more important 
examples. A complete list is not necessary,) 

arising #1 

Same format as above* 

8, Please tell us about short communications (memos, conversations, 
etc.) you have had which incorporated PC-based information ♦ (We are 
only interested in the more important exariples. A complete list is 
not necessary. 

Communication #1 



Same format as Above. 
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LARGER ORGANIZATIONAL CHANGES 

Source: Blkson et al.. Protocol for Managers 

1. Think about the changes in the department's work in relation to the 
organization. What changes have there been in work flow in/out of 
this group in relation to other departments? 

2. Have there been any changes in management styles and procedures? 
(Probe for keeping track of performance by computer and for number 

of people they can supervise.) 

3. Has planning or decision making become more decentralized or 
department centered, or has it becorca more centralized, from the top 
down? 

4. What do you think will be the net effect of computerization on the 
size and composition of the staff in this department? In the 
organization? (Probe for differences in sex ratios or age 
distribution.) 

5. If sizable labor force shifts or reductions are anticipated, how 
will they be handled? 

Source: Bikson 3t. al.. Protocol for Users 

6. Have there been any management innovations as a result of the 
technology? (e.g. quality circles?) 

Spuyce; Fleischer and Morell m 

7. Given all the times you can remember ever personally using office 
automation in your present job, what use had the greatest impact for 
you or your company? 

8. Why was the use described in question #7 so important? 

9. Given all the times you know of where your supervisor has personally 
used office automation equipment, what use do you believe had the 
greatest impact for your supervisor or the company? 

10. Why was the use described in question #9 so important? 

Note : Questions 9 and 10 are repeated relative to a person 
supervised by the respondent. 

11. The advent of office automation has given rise to claims that it 
affects organizational dynamics such as supervisory relationships, 
getting one's way with peers or superiors, coalition formation, 
communication patterns, and the like. In your experience, is any of 
this speculation true? YES NO 
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(If NO. skip to question # 1 
12. What organizational consequences of office automation have you 
observed? 



13, Do you have any pet theories about the effects of office automation 
on your company or the people who work there? If so, please share 
them with us. 
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